@@ Overview

@@ Getting started

@@ Base concepts

@@ C++ Instrumentation API

This chapter describes the C++ API of the Palanteer instrumentation library.

Initialization

The service shall be initialized once, before any usage of event tracing.
Server address and filename values shall be configured before the initialization function plInitAndStart.

plSetFilename

This function sets the record file path when in “file storage” mode. The path is copied, and its maximum size is 256 bytes.

To be taken into account, it shall be called before plInitAndStart function.

The default value is “record.pltraw”.

The declaration is:

// Sets the record file path when in "file storage" mode
void plSetFilename(const char* filename);

plSetServer

This function sets the server IP address and port when in “connected” mode (TCP socket).

It shall be called before plInitAndStart function to be taken into account.

The default value is “127.0.0.1" (IPv4 localhost) on port 59059.

The declaration is:

// Sets the server IP address and port when in "connected mode"
void plSetServer(const char* serverAddr, int serverPort);

plInitAndStart

This function initializes and starts the Palanteer service. It takes up to 4 parameters:

• the name of the application (mandatory)
  • ex: “Pacman”, “Space invaders”
• the event tracing mode: PL_MODE_CONNECTED, PL_MODE_STORE_IN_FILE and PL_MODE_INACTIVE
• an optional build name, to describe more precisely the version of the program
• an optional timeout (ms) to configure the duration that the program shall wait for the connection to the server before giving up.
  • the value 0 means infinite waiting
  • the value -1 means no wait (default)

The three modes of the event tracing are:

• PL_MODE_CONNECTED (default): connect to the server to enable remote recording and program control.
  • If waitForServerConnection is true, the initialization waits indefinitely for the established connection.
  • If waitForServerConnection is false (default), one connection to the server is tried. If it fails, the mode falls back to PL_MODE_INACTIVE.
  • The used parameters are the server and port values, configured with plSetServer
• PL_MODE_STORE_IN_FILE: the raw record is directly written in a file without any external connection.
  • The “raw record” data is in fact the data that would have been sent to the server.
  • In this mode, the remote control is inactive (no server...)
  • The used parameter is the storage filename, configured with plSetFilename.
• PL_MODE_INACTIVE: event collection and remote control are inactive, even if the Palanteer code is present

On top of these parameters, its behavior depends also on the configuration of the compilation flags:

• If USE_PL is not equal to 1, the function plInitAndStart does nothing at all.
  
• If USE_PL is 1, the behavior of this initialization function is:
  • if PL_IMPL_CATCH_SIGNALS is 1, it installs the signal handlers
  • if PL_IMPL_STACKTRACE is 1, it initializes the symbol decoder in case of crash (Windows only)
  • if the mode is set to PL_MODE_INACTIVE at run-time or both PL_NOEVENT and PL_NOCONTROL are set to 1 (compile-time disabling), the function returns here.
  • if PL_NOEVENT is not 1, the transmission thread is created and the context switches are initialized if enabled (compile time switch and enough run-time privilege)
  • if PL_NOCONTROL is not 1, the command reception thread is created.

The declaration is:

enum plMode { PL_MODE_CONNECTED, PL_MODE_STORE_IN_FILE, PL_MODE_INACTIVE};

// Initializes and starts the events and remote control services
void plInitAndStart(const char* appName, plMode mode=PL_MODE_CONNECTED, const char* buildName=0, int serverConnectionTimeoutMsec=0);

Details on the created threads are shared in the collection mechanism section.

Almost all memory allocations of the Palanteer services are done in this function. For details, refer to the section memory usage.

plStopAndUninit

This function stops and uninitializes the Palanteer service. It is typically called before exiting the program.

• It flushes to the server the last collected events, up to the call
  • the last events are often critical for a debugging investigation
• It properly stops the Palanteer threads
• It cleans the resources (i.e. threads and memory)

The declaration is:

// Stops and de-initialized the Palanteer service
void plStopAndUninit();

plGetStats

This function returns statistics about the collection process.

It can be called at any time.

The declaration is:

// Collection statistic structure
struct plStats {
    uint32_t collectBufferSizeByteQty;     // Configured collection buffer size
    uint32_t collectBufferMaxUsageByteQty; // Maximum used size in the collection buffer
    uint32_t collectDynStringQty;          // Configured dynamic string qty
    uint32_t collectDynStringMaxUsageQty;  // Maximum used dynamic string qty
    uint32_t sentBufferQty;                // Buffer qty sent to the server
    uint32_t sentByteQty;                  // Byte qty sent to the server
    uint32_t sentEventQty;                 // Event qty sent to server
    uint32_t sentStringQty;                // Unique string qty sent to server
};

// Get the collection statistics
plStats plGetStats(void);

Refer to the double storage bank mechanism description for more details.

Structure tracing

The instrumentation in this chapter traces structural elements to form the hierarchy of the collected data.
The timings are contained in this structure.

plDeclareThread

A thread can be given a name, at any moment during the recording, so that:

• it is easier to recognize
• it has a persistent identifier which makes scripting more reliable

The default name is "Thread N" with N the integer thread order of appearance (so may differ from run to run).

It has both group and dynamic string variants.

The declaration is:

// Thread name declaration with a static string name
void plDeclareThread(threadName);

// If the group is enabled, declares a thread with the provided name as static string
void plgDeclareThread(const char* group, const char* threadName);

// Thread name declaration with a dynamic string name
void plDeclareThreadDyn(const char* nameFormat, ...);

// If the group is enabled, declares a thread with the provided name as dynamic string
void plgDeclareThreadDyn(const char* group, const char* nameFormat, ...);

Some example of usage:

// Example 1
plDeclareThread("Worker 1");

// Example 2
plDeclareThreadDyn(threadName.c_str());

// Example 3
plDeclareThreadDyn("Worker %d", workerIdx);

Threads can also be clustered by prefixing the thread name with a cluster name, separated with "/" (similar to UNIX pathname).
Only one hierarchical level is accepted.

Some example of thread cluster:

// Important: Calls here are purely for explanation. plDeclareThread shall of course be called in its respective thread

// Standalone thread, not in a cluster
// (Called in a thread)
plDeclareThread("Render");


// The calls below implicitly define a cluster "Workers" which contains 3 threads

// (Called at the start of another thread)
plDeclareThread("Workers/Worker 1");

// (Called at the start of yet another thread)
plDeclareThread("Workers/Worker 2");

// (Called at the start of yet another thread)
plDeclareThreadDyn("Workers/Worker %d", workerIdx);

plScope

This function defines a scope with the provided name.

A “scope” is a named time range with a start and an end, which can contains children (scopes or data).

• The start of the scope is the call of this function plScope
• The end of the scope corresponds to the end of the scope in the language

It has both group and dynamic string variants.

The declaration is:

// Declares a scope with the provided name as static string
void plScope(const char* name);

// If the group is enabled, declares a scope with the provided name as static string
void plgScope(const char* group, const char* name);

// Declares a scope with the provided name as dynamic string
void plScopeDyn(const char* name);

// If the group is enabled, declares a scope with the provided name as dynamic string
void plgScopeDyn(const char* group, const char* name);

plFunction

This function automatically declares a scope with the current function name.
It is equivalent to

plScope(__FUNCTION__);

The start of the scope is the function call, its end is the one of the language scope.

It has both group and dynamic string variants.

The declaration is:

// Declares a scope with the current function name as static string (check your compiler for support, see note below)
void plFunction(void);

// If the group is enabled, declares a scope with the current function name as static string
void plgFunction(const char* group);

// Declares a scope with the current function name as dynamic string
void plFunctionDyn(void);

// If the group is enabled, declares a scope with the current function name as dynamic string
void plgFunctionDyn(const char* group);

plBegin and plEnd

These two functions are used together, they define an explicit start and end of a scope.

The provided name of the scope shall be the same for both functions, so that any mismatch can be detected on the server side.

The declaration is:

// Declares the start of a scope with the provided name as static strings
void plBegin(const char* name);

// Declares the end of a scope with the provided name as static strings
// The name shall match the begin name, so mismatches can be detected on viewer side and lead to warnings
void plEnd(const char* name);

// If the group is enabled, declares the start of a scope with the provided name as static strings
void plgBegin(const char* group, const char* name);

// If the group is enabled, declares the end of a scope with the provided name as static strings
void plgEnd(const char* group, const char* name);

// Declares the start of a scope with the provided name as a dynamic string
void plBeginDyn(const char* name);

// Declares the end of a scope with the provided name as a dynamic string
// The name shall match the begin name, so mismatchs can be detected on viewer side and lead to warnings
void plEndDyn(const char* name);

// If the group is enabled, declares the start of a scope with the provided name as a dynamic string
void plgBeginDyn(const char* group, const char* name);

// If the group is enabled, declares the end of a scope with the provided name as a dynamic string
void plgEndDyn(const char* group, const char* name);

Data tracing

The instrumentation in this chapter traces string or data of any numeric type inside the current scope.

Such events can be visualized in the viewer under several forms: text, curve or histogram.
They can optionally be associated to a unit by ending the event name with ##<unit>:

• The unit is stripped from the name, it is not displayed
• The unit increases the semantic of the events and make is more useful for the users
• Only events with consistent units can be plotted together
• “hexa” is a special and hardcoded unit which displays integer in hexadecimal

Ex: "Duration##nanosecond", "pointer##hexa", "Banana##fruit"

plData

This function traces a named value.

The type of the value can be any integral types plus float, double and string (processed as dynamic string).

The declaration is:

// Traces a value
void plData(const char* name, <type> value);

// If the group is enabled, traces a value
void plgData(const char* group, const char* name, <type> value);

Example of usage:

int a = 15;
uint64_t b = 314159265359LL;
const char* c = "I am a dynamic string because not constexpr"; // Or just "char*"

// Various value tracing
plData("my value A", a);
plData("Truncated PI", 3.14159265359);
plData("Big PI", b);
plData("Dynamic string", c);

// Group variant
plgData(MY_DETAIL, "my value A", a); // PL_GROUP_MY_DETAIL must be defined with value 0 or 1

// Tracing of a static string, with the help of plMakeString
plData("Status", plMakeString("Low run time resource with such static string"));

// Another dynamic string message
const char* states[3] = { "Good automata state", "Average automata state", "Bad automata state" };
plData("State", states[i]);

// Compared to previous example, using plString_t reverts back to the static string performance and behavior
// Only the string selection is dynamic, the strings are processed and hashed at compile time.
plString_t static_states[3] = { plMakeString("Good automata state"), plMakeString("Average automata state"), plMakeString("Bad automata state") };
plData("State", static_states[i]);

plText

This function traces a named static string message.
It is a convenience function which calls plData with the plMakeString static string helper.

It has a group variant.

The declaration is:

// Traces a static string message
void plText(const char* name, const char* msg);

// If the group is enabled, traces a static string message
void plgText(const char* group, const char* name, const char* msg);

palanteer.h: error: "string variable" is not a constant expression
| #define PL_STRINGHASH(s) plPriv::forceCompileTimeElseError_ < plPriv::fnv1a_(s,PL_FNV_HASH_OFFSET_) > ::compileTimeValue
... (then a lot more)

Example of usage:

// Static string message
plText("Stage", "Reaching a critical point");

plVar

This function is a convenience to trace the content of one or more variables under their respective variable name.

The type of the variable can be any integral types plus float, double and string (processed as dynamic string).

The declaration is:

// Traces the value of variables under the name of the variables
void plVar(variable1, variable2 ...);

// If the group is enabled, traces the value of variables under the name of the variables
void plgVar(const char* group, variable1, variable2 ...);

Between 1 and 10 variables or expressions are accepted.

Example of usage:

unsigned short myFirstVariable = 1000;
const char*    aStringVariable = "I am a string";

// Trace variables
plVar(myFirstVariable, aStringVariable);
// Group version
plgVar(MY_DETAIL, myFirstVariable, aStringVariable); // PL_GROUP_MY_DETAIL must be defined beforehand with value 0 or 1


// Longer (strictly) equivalent tracing:
plVar(myFirstVariable);
plVar(aStringVariable);
plgVar(MY_DETAIL, myFirstVariable);
plgVar(MY_DETAIL, aStringVariable);

// Even longer (strictly) equivalent tracing:
plData("myFirstVariable", myFirstVariable);
plData("aStringVariable", aStringVariable);
plgData(MY_DETAIL, "myFirstVariable", myFirstVariable);
plgData(MY_DETAIL, "aStringVariable", aStringVariable);

Note that plVar works with any expression:

// "i" will be incremented by one as expected. However, not recommended *at all*, as "i++" is not called when Palanteer is disabled
plVar(i++);
// Any expression is ok
plVar(state[i], htons(i), atan(i*3.14159/17.));

// The two previous calls are strictly equivalent to:
plData("i++", i++);
plData("state[i]", state[i]);
plData("htons[i]", htons[i]);
plData("atan(i*3.14159/17.)", atan(i*3.14159/17.));

Logs

The instrumentation in this chapter refers to “usual logging”, defined as the union of the definition below:

• a message from the application
• formatted through a printf-like interface
• automatically timestamped
• with a user specified level
• with a user specified “category” (i.e. a name to help filtering)

In the viewer:

• a dedicated log window provides filtering capabilities on threads, levels and categories
• an indicator is visible for each log on top of the timeline for the associated thread (configurable level)

Four log levels are defined, in this order:

enum plLogLevel {
    PL_LOG_LEVEL_ALL   = 0,  // Just for code clarity
    PL_LOG_LEVEL_DEBUG = 0,
    PL_LOG_LEVEL_INFO  = 1,
    PL_LOG_LEVEL_WARN  = 2,
    PL_LOG_LEVEL_ERROR = 3,
    PL_LOG_LEVEL_NONE  = 4
};

plSetLogLevelRecord

This function dynamically sets the minimum log level to be recorded. All logs with a level strictly below the provided level are dynamically filtered out from the record.
The default value is PL_LOG_LEVEL_ALL.

The declaration is:

void plSetLogLevelRecord(plLogLevel level);

A level of PL_LOG_LEVEL_ALL (equivalent to PL_LOG_LEVEL_DEBUG) includes all logs in the record.

A level of PL_LOG_LEVEL_NONE removes all logs from the record.

plSetLogLevelConsole

This function dynamically sets the minimum log level to be displayed on console. All logs with a level strictly below the provided level are dynamically filtered out from the console display.
The default value is PL_LOG_LEVEL_WARN.

The declaration is:

void plSetLogLevelConsole(plLogLevel level);

A level of PL_LOG_LEVEL_ALL (equivalent to PL_LOG_LEVEL_DEBUG) displays all logs on the console.

A level of PL_LOG_LEVEL_NONE does not display any log on the console.

The configuration flag PL_IMPL_CONSOLE_COLOR is used by the log console display.

plLogDebug

This function logs a printf-like message with a category and a debug level, automatically timestamped.
The purpose of the “category” attribute is to offer filtering per user-defined topic.

All standard printf argument formats are supported (except the strange and non remote compatible %n), and all provided parameters can be graphed individually.

This function has the group variant (note that the category and the format strings are always static strings).

The declaration is:

// Log a message
void plLogDebug(const char* category, const char* format, ...);

// If the group is enabled, log a message
void plgLogDebug(const char* group, const char* category, const char* format, ...);

Example of usage:

plLogDebug("input", "Key '%c' pressed", pressedKeyChar);

plLogDebug("output", "The resulting value of the phase %-20s is %g with the code 0x%08x", phaseStr, floatResult, errorCode);

plgLogDebug(MY_DETAIL, "phase", "End of a computation"); // PL_GROUP_MY_DETAIL must be defined with value 0 or 1

plLogInfo

This function is similar to the plLogDebug function but logs with the level info.

plLogWarn

This function is similar to the plLogDebug function but logs with the level warn.

plLogError

This function is similar to the plLogDebug function but logs with the level error.

plMarker (deprecated)

Markers are superseded by the previous log APIs, which are a kind of generalization. The plMarker APIs will be removed in a future release.

At the moment, they are mapped to a call to plLogWarn.

Lock tracing

The instrumentation in this chapter traces actions performed on locks (as a general term).

The usage of locks in multithreaded environment can deeply modify the dynamic behavior.
Visualizing them becomes critical in complex programs, that is why a specific diagram in the viewer helps identifying the bottlenecks across threads.

The lock API is “low level” so that it adapts to many existing lock primitives (mutex, semaphores, std::unique_lock, condition variables...), at the price of some less automatic instrumentation work.
This instrumentation becomes much lighter if the program uses an OS abstraction layer, as only this layer needs to be instrumented.

The dynamic variant allows to provide dynamic lock name string, at the cost of some extra runtime computations (one copy and one hashing).
The static variant allows to provide a static lock name through a function without extra runtime cost, which is convenient in some cases of wrapper implementation. It uses the structure plString_t.

Some examples of instrumentation of several usual primitives are shown after the description of the lock API

plLockWait

The locking process usually starts with waiting for the lock.
This function traces this first step.

When the wait ends, plLockState(...) or plLockScopeState(...) must be called to set both:

• the end of the waiting phase
• the state of the lock: true if the lock is taken, or false if not.

This function has both group and dynamic string variants.

The declaration is:

// Set the start of waiting for a lock
void plLockWait(const char* name);

// If the group is enabled, set the start of waiting for a lock
void plgLockWait(const char* group, const char* name);

// Set the start of waiting for a lock which has a static name represented by a plString_t
void plLockWaitStatic(plString_t name);

// If the group is enabled, set the start of waiting for a lock which has a static name represented by a plString_t
void plgLockWaitStatic(const char* group, plString_t name);

// Set the start of waiting for a lock which has a dynamic name
void plLockWaitDyn(const char* name);

// If the group is enabled, set the start of waiting for a lock which has a dynamic name
void plgLockWaitDyn(const char* group, const char* name);

Example of usage for a pthread mutex lock:

plLockWait("Database access");        // Start waiting for the lock named "Database access"
pthread_mutex_lock(&_mutexDb);
plLockState("Database access", true); // >>> Do not forget this call! <<<  Lock is taken (see next section)
...

plLockState

This function is involved in both phases of the locking process.
Its main role is to set the state of the lock usage:

• true state means the lock is taken by the thread
• false state means the lock is released (or not used) by the thread

Its other and implicit role is to mark the end of the waiting process (if it was started previously), whatever the lock state value.

This function has both group and dynamic string variants.

The declaration is:

// Set the lock state
void plLockState(const char* name, bool lockUseState);

// If the group is enabled, set the lock state
void plgLockState(const char* group, const char* name, bool lockUseState);

// Set the lock state when the lock has a static name represented by a plString_t
void plLockStateStatic(plString_t name, bool lockUseState);

// If the group is enabled, set the lock state when the lock has a static name represented by a plString_t
void plgLockStateStatic(const char* group, plString_t name, bool lockUseState);

// Set the lock state when the lock has a dynamic name
void plLockStateDyn(const char* name, bool lockUseState);

// If the group is enabled, set the lock state when the lock has a dynamic name
void plgLockStateDyn(const char* group, const char* name, bool lockUseState);

Example of usage for a pthread mutex unlock:

...
// Lock is released
plLockState("Database access", false); // Lock is released (trace before the release call)
pthread_mutex_unlock(&_mutexDb);

plLockScopeState

This function is required when the “unlock” is triggered by RAII (i.e. automatically at the end of the language scope).
It is a concatenation of an immediate plLockState call to set the lock state typically after a lock wait and an automatic call to plLockState(false) (aka unlock) at the end of the language scope.

It typically matches the behavior of std::unique_lock (and variants).

This function has both group and dynamic string variants.

The declaration is:

// Set the lock state and automatically unlocks at the end of the scope
void plLockScopeState(const char* name, bool lockUseState);

// If the group is enabled, set the lock state and automatically unlocks at the end of the scope
void plgLockScopeState(const char* group, const char* name, bool lockUseState);

// Set the lock state when the lock has a dynamic name and automatically unlocks at the end of the scope
void plLockScopeStateDyn(const char* name, bool lockUseState);

// If the group is enabled, set the lock state when the lock has a dynamic name and automatically unlocks at the end of the scope
void plgLockScopeStateDyn(const char* group, const char* name, bool lockUseState);

Example of usage for std::unique_lock (or std::lock_guard here):

{
    plLockWait("Resource");  // Start waiting for the lock "Resource"
    std::unique_lock<std::mutex> lk(globalCriticalResourceMutex, std::try_to_lock);
    plLockScopeState("Resource", lk.owns_lock()); // Stop the wait and set the state of the lock
    ...
} // If previously taken, the "unlock" state is automatically logged here at the end of the scope

plLockNotify

This function traces a notification or a “post” on a lock.
It has no direct effect on the lock state.

It typically matches the behavior of a condition_variable notify_one() call, or a semaphore post.

This function has both group and dynamic string variants.

The declaration is:

// Trace a lock notification
void plLockNotify(const char* name);

// If the group is enabled, trace a lock notification
void plgLockNotify(const char* group, const char* name);

// Trace a lock notification when the lock has a static name represented by a plString_t
void plLockNotifyStatic(plString_t name);

// If the group is enabled, trace a lock notification when the lock has a static name represented by a plString_t
void plgLockNotifyStatic(const char* group, plString_t name);

// Trace a lock notification when the lock has a dynamic name
void plLockNotifyDyn(const char* name);

// If the group is enabled, trace a lock notification when the lock has a dynamic name
void plgLockNotifyDyn(const char* group, const char* name);

Example of usage:

std::string lockName("Task wait");

if(shallWakeTheTask) {
    plLockNotifyDyn(lockName.c_str()); // Notify the "Task Wait" lock (linked to a condition variable)
    taskWaitConditionVariable.notify_one();
}

Examples of lock instrumentation

Example with pthread mutex:

pthread_mutex_t resultsLock;
pthread_mutex_init(&resultsLock, NULL); // Ok, we should check the returned value...

plLockWait("Result");                        // Start of the lock wait
if(pthread_mutex_lock(&resultsLock)!=0) {
    plLockState("Result", false);            // End of the lock wait and lock not taken
    return;
}
plLockState("Result", true);                 // End of the lock wait and lock taken

...

plLockState("Result", false);                // End of the lock usage, lock released
pthread_mutex_unlock(&_resultsLock);

Example with pthread semaphore:

sem_t sem;
sem_init(&sem, 0, 1);

// Thread 1 (displayer client)
while(1) {
    plLockWait("Wake");           // Start of the lock wait
    sem_wait(&sem);
    plLockState("Wake", false);   // End of the lock wait and no lock taken (only the wait matters for this semaphore)

    // Do something
    printf("Some work\n");
}


// Thread 2 (controller)
while(1) {
    plLockNotify("Wake");         // Send a notification for the lock. It shall indirectly stop the semaphore wait phase
    sem_post(&sem);

    // Wait a bit
    useconds_t r = random() % 100;
    usleep(r);
}

Example with std::mutex:

std::mutex database_mutex;
std::map<std::string, std::string=""> databse;

void addUrl(const std::string& url, const std::string& result)
{
    plLockWait("Database");                            // Start waiting for the lock "Database"
    std::lock_guard<std::mutex> guard(database_mutex);
    plLockScopeState("Database", true);                // End of the lock wait and lock taken each time by design
    database[url] = result;
} // Automatic trace of the unlock because we used plLockScopeState()

Example with std::condition_variable:

std::mutex mtx;             // mutex for critical section
std::condition_variable cv; // condition variable for critical section
bool doWakeUp = false;

// Thread 1 (displayer client)
while(1) {

    plLockWait("Wake");                             // Start of the scoped lock wait
                                                    // Note: sometimes, tracking the wait is not interesting. This line can then be omitted.
    std::unique_lock<std::mutex> lk(mtx);           // Locking the mutex of the condition variable
                                                    // The mutex is taken here, but this is not the end of the "condition variable waiting phase"
    cv.wait(lk, [&doWakeUp] { return doWakeUp; });  // Waiting until the condition is true
    plLockScopeState("Wake", true);                 // Lock always taken by design (with automatic RAII unlock)

    // Do something
    printf("Some work\n");

    // Reset the wake up
    doWakeUp = false;

} // Automatic trace of the unlock because we used plLockScopeState()


// Thread 2 (controller)
while(1) {

    // Wait a bit
    useconds_t r = random() % 100;
    usleep(r);

    // Make the wake up condition true
    std::unique_lock<std::mutex> lk(mtx);
    doWakeUp = true;
    plLockNotify("Wake");                          // Send a notification for the lock, which shall indirectly break the condition variable waiting phase
    cv.notify_one();
}

CLIs

“CLIs” (Command Line Interface) are functions inside the instrumented program that can be called remotely and with parameters.

These functions, also named “CLI handlers”, are called from the Palanteer reception thread.

plRegisterCli

This function registers a CLI.
It requires a CLI handler, the name of the CLI, the CLI parameter specification and a user description.

The name of the command shall not contain spaces. The user description is purely for documentation purpose. These two strings must be static and are affected by the “external string” features.
The parameter specification must also be a static string, but it is never obfuscated because its content is used internally to validate the command syntax.

The declaration is:

void plRegisterCli(plCliHandler_t cliHandler, const char* name, const char* param_specification, const char* description);

An example of CLI registration is (see Remote control for full example):

plRegisterCli(setBoundsCliHandler, "config:setRange", "min=int max=int", "Sets the value bounds of the random generator");

CLI handler

A CLI handler is a function triggered remotely that interacts with the controlled program.
It accepts typed parameters among integer, float or string and returns a status and a text.

The communication with Palanteer is done through a “communication helper” object. Its prototype is:

typedef void (*plCliHandler_t)(plCliIo& cio);

This “communication” object has two roles:

• Provide the input parameters via typed accessors
  • These parameters are already checked for consistency versus the CLI parameter specification
  • Using the wrong accessor type is a bug and leads to an assertion failure.
• Format the command output, namely the binary success state and the text response
  • Initial response is empty and texts provided by addToResponse are concatenated.
  • A call to setErrorState indicate an execution failure (non-cancellable)
    • Previous text response is cleared
    • Some text can be provided directly in the call. Subsequent calls to addToResponse add up.

Its full prototype is:

class plCliIo {
public:
    // Input accessors
    int64_t     getParamInt   (int paramIdx) const;
    double      getParamFloat (int paramIdx) const;
    const char* getParamString(int paramIdx) const;

    // Output formatting
    void setErrorState(const char* format=0, ...); // Set the error state and take some optional text
    bool addToResponse(const char* format, ...);   // Returns false if the response buffer is full
    void clearResponse(void);                      // Resets the response buffer

An example of CLI handler extracted from Remote control and declared with the parameter spec "min=int max=int" is:

// Handler (=implementation) of the example CLI
void setBoundsCliHandler(plCliIo& cio)  // 'cio' is a communication helper present in each handler call
{
    int minValue = cio.getParamInt(0); // Get the 2 CLI parameters as integers
    int maxValue = cio.getParamInt(1);
    if(minValue>maxValue) {
        // CLI execution has failed. The text answer contains some information about it
        cio.setErrorState("Minimum value (%d) shall be lower than the maximum value (%d)\n", minValue, maxValue);
        return;
    }

    // Modify the state of the program
    globalMinValue = minValue;
    globalMaxValue = maxValue;
    // CLI execution was successful (no call to cio.setErrorState())
}

And a way to call it from a Python program with the Palanteer scripting module is:

palanteer.program_cli("config:setRange min=300 max=500")

Virtual threads

A “virtual thread” is a thread that is not managed by the OS. Some examples are “fibers”, or a simulated OS environment.
They require the support of “OS worker threads” which effectively run these virtual threads. A running virtual thread can be switched/exchanged with another one usually only at particular points (I/O call, yield call, etc...), and resumed later on any of the existing worker threads.

The support of virtual threads requires the following actions:

• Use the compilation option PL_VIRTUAL_THREADS=1 (in all files)
• Notify Palanteer of any virtual thread switch through the plAttachVirtualThread and plDetachVirtualThread API
  • this notification shall be called inside the assigned worker thread for proper association between the virtual thread and the OS worker thread
  • typically in the virtual thread switch hook of the virtual thread framework

Optionally but recommended, the virtual thread name can be declared with plDeclareVirtualThread, typically in the virtual thread creation hook.

As an example, the Windows function SwitchToFiber call shall be preceded by a plDetachVirtualThread call and followed by a plAttachVirtualThread call.
The available information shall be if the “old” job is suspended for later resume or finished, and after the switch the ID of the new fiber.

The effects of using virtual threads are:

• Each virtual thread is seen as a “normal” thread on the server side (viewer or scripting)
• All events generated during the execution of a virtual thread are associated to this virtual thread, not to the OS thread.
  • in the viewer, an interruption of the execution of a virtual thread is indicated as a “SOFTIRQ Suspend” for this thread
• Worker threads (OS thread) look “empty”
  • only the CPU context switches, if enabled, are associated with them
• To get an usage overview, a “resource” with the name of each worker thread tracks slices of its time used by virtual threads.

plDeclareVirtualThread

As for OS threads, a name can be given to virtual threads.
This function associates the provided name to the external virtual thread identifier. The name of the OS thread is unchanged.

This function can be called from any thread. Only the unique virtual thread identifier matters here.

The declaration is:

// externalVirtualThreadId: a unique external virtual thread identifier
// name: the name of the virtual thread
void plDeclareVirtualThread(uint32_t externalVirtualThreadId, const char* nameFormat, ...);

Example of usage:

plDeclareVirtualThread("Fibers/Fiber %d", fiberId);

Also as for OS threads, virtual threads can be grouped.

plAttachVirtualThread

This function notifies Palanteer of a virtual thread attachment to the current OS thread.

The declaration is:

// externalVirtualThreadId: a unique external virtual thread identifier
void plAttachVirtualThread(uint32_t externalVirtualThreadId);

Example of usage:

plAttachVirtualThread(newFiberId);

plDetachVirtualThread

This function notifies Palanteer that the current virtual thread is detached from the current OS thread.

The declaration is:

// isSuspended: true if this new virtual thread ID is suspended, false if it completed his previous task
void plDetachVirtualThread(bool isSuspended);

Example of usage:

plDetachVirtualThread(false);

Troubleshootings

I get a lot of preprocessor errors...

The usage of cascaded macros in Palanteer, only practical way to do some introspection in C++, has the bad effect of creating a cascade of errors...
In such case, try to isolate the faulty line from your sources, and check:

• that the group is indeed defined, if you use groups
• that the string is really static, if you use an API which requires a static string
• that the logged variable is indeed one of the traceable type (i.e. C-string or a numerical value)

I really get a lot of preprocessor errors while using logging, talking about ambiguous overload stuff...*

First, apologies for the quantity of thrown error, see the previous point.
This problem is independent of the string format. It is purely related to the provided parameters.

Of course, check first that the parameters can all be handled properly by printf (no struct, class, ...).
Then the issue is probably due to the fixed width integer types, especially for 64 bits integers:

• Palanteer handles the non ambiguous definition (from C++11) int8_t, uint8_t, int16_t, uint16_t, int32_t uint32_t, int64_t and uint64_t
• these types do not correspond always to the “old” types. For instance, long long int is not always a int64_t
• some compilers get crazy when having to deal with such casts, hence the overload ambiguity

To solve the issue, just cast the faulty parameter with its corresponding fixed width integer types (ex: (int64_t)myLongLongInt)

plFunction does not compile

You probably hit the issue described in plFunction: some “old” C++ compilers do not consider __FUNCTION__ as constexpr.
In such case, either:

• switch to plFunctionDyn(), with the drawback of the non optimal dynamic string management
• use plScope("manually copied function name")
• use a more recent compiler, if possible

I use clang with ASAN and the memory allocations are not logged...

Overloading new and delete operators in clang with ASAN does not work, this is a known issue in clang (see clang bugzilla https://bugs.llvm.org/show_bug.cgi?id=19660 ).

I have some unexpected crash in the instrumentation library

Ensure that the non-implementation configuration flags are consistent in all files (PL_EXTERNAL_STRINGS, PL_DYN_STRING_MAX_SIZE, ...).
This can really lead to undefined behaviors.

I traced some data but the viewer reports it as an instrumentation error

Data events (text, numerical value) are not timestamped.
Outside a time scope, which means only at the root level of the tree, they cannot be associated to any dated element (the knowledge that they are after the last one and before the next one is not used).
The fix is to move them inside a scope, or create a scope to hold them.

Note that printf-like logs and lock notifications do not have this constraint and can be located at the root level because they contain a timestamp.

What is the name of the function to break in when investigating a crash under a debugger?

Break in plCrash.

I have many threads in parallel and many “SATURATION” logs despite I allocated enough memory for the event collection buffers

If all CPUs are saturated, the collection task cannot run regularly enough and buffers will get full whatever their allocated size.
The consequence is that event tracing will start to block the threads, waiting for some space for tracing, and place a “SATURATION” error log.

The presence of the viewer or scripting module on the same machine is maybe the problem, as they use also some CPU and may interfere with the program under observation.
Ideally, the available CPU quantity shall be: at least the required quantity for your program + 3 (instrumentation thread + viewer recording thread + viewer display thread).

Recording on a file reduces this requirement to: at least the required quantity for your program + 1 (instrumentation thread), as the processing of the raw events will be done later.

What are the limitations on thread and event quantity?

The system can handle up to 254 threads and 2 billion events.
The estimated size of the record would be ~20 GB, still displayed smoothly.

I want to hide the Palanteer threads

The whole instrumentation of the implementation of the Palanteer threads is using the group PL_VERBOSE.
It can be fully disabled with the line below, to be inserted before the #include "palanteer.h" which implements the service:

#define PL_GROUP_PL_VERBOSE 0

@@ C++ Instrumentation configuration

@@ Python Instrumentation API

@@ Scripting API

@@ More