sslog: Speedy Structured C++17 logging library

sslog: Speedy Structured C++17 logging library
Go to code

Contents

(Top)
Overview
  1.1  Features
  1.2  Install
  1.3  Benchmarks
  1.4  Examples
    1.4.1  Basic logging with names and units
    1.4.2  Details on demand
    1.4.3  Query with “sscat” shell tool
    1.4.4  Structured output in JSON with “sscat” shell tool
    1.4.5  Query in Python
    1.4.6  Query in C++
    1.4.7  Binary buffer logging
    1.4.8  Log notification callback
    1.4.9  Multisink with fully configurable console display
    1.4.10  Rotating files
    1.4.11  Selection of groups of log at compile-time
  1.5  FAQ
  1.6  Dependencies
  1.7  License
Logging configuration
Logging reference API
sslogread Python module
sslogread C++ API
sscat

  

@@ Overview

Speedy Structured logging library

Speedy Structured C++17 logging library

To achieve 30 million logs per second, string information is pre-processed at compile-time so that practically only arguments are stored at run-time.

Furthermore, compact storage is obtained by writing duplicate strings only once, in binary form and with generalized delta-encoding.

Costly formatting is deferred to logs exploitation phase, enabling dynamic filtering, queries and transforms:
Logs act as a tiny database.

The flexible query interface is available in shell, python or C++, and is suitable for


In including just one header file.

   

Features

   

Install

Copy and include the single header sslog.h.
Then start logging!

The easiest way to install the python reader module is:

python3 -m pip install sslogread
   

Benchmarks

Raw results from the internal benchmark (run test/run_suite.py -s performance):

==============================================================================
| sslog - Logging runtime 0 param             | 26.0 ns or 38.46 Mlog/s      |
| sslog - Logging runtime 1 param int         | 28.3 ns or 35.39 Mlog/s      |
| sslog - Logging runtime 1 param string      | 33.2 ns or 30.11 Mlog/s      |
| sslog - Logging runtime 4 params int        | 33.9 ns or 29.51 Mlog/s      |
| sslog - Logging runtime skipped log         | 0.1 ns or 13543.67 Mlog/s    |
|                                             |                              |
| sslog - compactness - full text             | 100% (1e6 logs / 151.9 MB)   |
| sslog - compactness - sslog         vs text | 11.9 %                       |
| sslog - compactness - sslog+zstd -5 vs text | 3.0 %                        |
|                                             |                              |
| sslog - Header include                      | 0.703 s                      |
| sslog - Header include - SSLOG_DISABLE=1    | 0.243 s                      |
|                                             |                              |
| sslog - Library code size (-O2)             | 45208 bytes                  |
| sslog - Log code size 0 param (-O2)         | 60 bytes/log                 |
| sslog - Log code size 1 param int (-O2)     | 80 bytes/log                 |
| sslog - Log code size 1 param string (-O2)  | 72 bytes/log                 |
| sslog - Log code size 4 params int (-O2)    | 112 bytes/log                |
|                                             |                              |
| sslog - Log compilation speed (-O0)         | 6578 log/s                   |
| sslog - Log compilation speed (-O2)         | 914 log/s                    |
==============================================================================

Highlights:

Benchmark setup

See comparison with spdlog

Raw results from the internal benchmark (run test/run_suite.py -s performance_spdlog with spdlog-dev installed):

==============================================================================
| spdlog - Logging runtime 0 param            | 344.3 ns or 2.90 Mlog/s      |
| spdlog - Logging runtime 1 param int        | 349.0 ns or 2.87 Mlog/s      |
| spdlog - Logging runtime 1 param string     | 350.1 ns or 2.86 Mlog/s      |
| spdlog - Logging runtime 4 params int       | 410.6 ns or 2.44 Mlog/s      |
| spdlog - Logging runtime skipped log        | 2.5 ns or 392.63 Mlog/s      |
|                                             |                              |
| spdlog - Header include                     | 3.035 s                      |
|                                             |                              |
| spdlog - Library code size (-O2)            | 241840 bytes (+ shared libs) |
| spdlog - Log code size 0 param (-O2)        | 20 bytes/log                 |
| spdlog - Log code size 1 param int (-O2)    | 56 bytes/log                 |
| spdlog - Log code size 1 param string (-O2) | 56 bytes/log                 |
| spdlog - Log code size 4 params int (-O2)   | 144 bytes/log                |
|                                             |                              |
| spdlog - Log compilation speed (-O0)        | 11008 log/s                  |
| spdlog - Log compilation speed (-O2)        | 2447 log/s                   |
==============================================================================

Highlights:

Benchmark setup

  • spdlog version 1.12 (from Ubuntu 24.04.2 LTS), compiled version
  • Configuration similar to sslog:
    • Asynchronous logger, thread-safe sink basic_file_sink_mt
    • 1 thread pool with queue size = 65535, block overflow policy, no console
  • Runtime performance measured on 1 million logs up to storage

   

Examples

   

Basic logging with names and units

#include "sslog.h"

int main()
{
    ssInfo("myApp/main", "First message on console with sslog!");
    ssError("myApp/main", "First error with parameters: %d %-8s %f %08lx",
            14, "stream", 3.14, 1234567890UL);

    // The optional text before and after an argument is interpreted as name and unit
    // The category (first string) aims typically user-specific classification
    ssDebug("/animal/pet/dog", "surname=%s weight=%4.1f_kg", "mirza", 12.856);
    ssTrace("/animal/pet/cat", "surname=%s weight=%4.1f_kg", "misty", 5.5);
}

See here for details on the logging API.

   

Details on demand

A unique feature is the logging of details upon request.

In the example below:

#include "sslog.h"

int main()
{
    // Configure the level of the details and the file split rules
    sslog::Sink config;
    config.storageLevel = sslog::Level::info;  // Standard logging at "info" level
    config.detailsLevel = sslog::Level::trace; // On request, enables details up to 'trace' (for storage)
    config.fileMaxBytes = 10000;               // Granularity of 10 KB
    ssSetSink(config);

    for (int iter = 0; iter < 100000; ++iter)
    {
        ssInfo("Always logged and stored");
        ssDebug("Logged and stored only around the iteration 50000. Current iteration is %05d", iter);

        // Request details up to 'trace' before and after this particular moment
        // The duration of the surrounding context capture is configurable
        if (iter == 50000) ssRequestForDetails();
    }
}
   

Query with “sscat” shell tool

Display all the logs (no filtering) in text:

sscat <log dir>

Select the logs on a category matching *engine* but not on a thread matching worker*

sscat <log dir> -nt "worker*" -c "*engine*"

Select the logs on category transaction and which possess an argument named id which has a value higher than 356 and lower or equal to 1000

sscat <log dir> -c "transaction" -a "id>356" -a "id<=1000"

See here for details.

   

Structured output in JSON with “sscat” shell tool

The option -j switches to the JSON output:

sscat <log dir> -j
   

Query in Python

Display all the logs (no filtering) in text:

import sslogread
session = sslogread.load("/path/to/my/log_folder")

result = session.query()
for log in result:
   print("%d) %s" % (log['timestampUtcNs'], log['format'])) # Simple display

Select the logs on a category matching *engine* but not on a thread matching worker*

result = session.query( { 'no_thread': "worker*", 'category':'*engine*' } )

Select the logs on category transaction and which possess an argument named id which has a value higher than 356 and lower or equal to 1000

result = session.query( { 'category': "transaction", 'arguments': ["id>356", "id<=1000"] } )

See here for details on the Python reader module.

   

Query in C++

Display all the logs (no filtering) in text:

std::string errorMessage;
sslogread::LogSession session;
if(!session.init("/path/to/my/log/folder", errorMessage)) {
    fprintf(stderr, "Error: %s\n", errorMessage.c_str());
    exit(1)
}

if(!session->query({}, [session](const sslogread::LogStruct& log) {

                 // Format the string with arguments (=custom vsnprintf with our argument list, see below)
                 char filledFormat[1024];
                 sslogread::vsnprintfLog(filledFormat, sizeof(filledFormat), session->getIndexedString(log.formatIdx), log.args, session);

                 // Some simple display on console (there are better ways)
                 printf("[timestampUtcNs=%lu  thread=%s  category=%s  buffer=%s] %s\n",
                     log.timestampUtcNs, session->getIndexedString(log.threadIdx), session->getIndexedString(log.categoryIdx),
                     log.buffer.empty()? "No" : "Yes", filledFormat);

             },
             errorMessage)
{
    fprinf(stderr, "Error: %s\n", errorMessage.c_str());
}

Select the logs on a category matching *engine* but not on a thread matching worker*

Rule rule;
rule.category = "*engine*";
rule.noThread = "worker*";

std::string errorMessage;
if(!session.query({rule}, [](const sslogread::LogStruct& log) { /* ... */ }, errorMessage)
{
    fprinf(stderr, "Error: %s\n", errorMessage.c_str());
}

Select the logs on category transaction and which possess an argument named id which has a value higher than 356 and lower or equal to 1000

Rule rule;
rule.category = "transaction";
rule.arguments.push_back("id>356");   // Note: comparison works also with strings (alphanumerically)
rule.arguments.push_back("id<=1000");

/* ... */

See here for details on the C++ reader API.

   

Binary buffer logging

The printf interface provides built-in format checks,is wildly known and is quite expressive.
In this cotext, inserting support of custom structure cannot be easily supported.

An exception is made for the valuable case of generic binary buffers (dumps, images, packets, CAN messages...).

The API is similar to sslog standard logging with the suffix “Buffer” and with additional arguments: buffer pointer and size before the format string.

#include "sslog.h"
#include <vector>

int main()
{
    ssSetConsoleLevel(sslog::Level::trace);

    std::vector<uint8_t> buffer{0xDE, 0xAD, 0xBE, 0xEF};

    ssDebugBuffer("myApp/archive", buffer.data(), buffer.size(),
                  "Standard log with a binary buffer attached. Its sweet name is %s, id:%d",
                  "Francis", 314);
}

See here for details.

   

Log notification callback

A custom log handler, or some special processing on high level logs?
Simply register a callback and set the start level.

#include "sslog.h"

int main()
{
    sslog::Sink config;
    config.liveNotifLevel = sslog::Level::critical;
    config.liveNotifCbk = [] (uint64_t timestampUtcNs, uint32_t level, const char* threadName, const char* category,
                              const char* logContent, const uint8_t* binaryBuffer, uint32_t binaryBufferSize)
                              {
                                  sendEmergencyMail("support@world.com", timestampUtcNs, category, logContent);
                              };
    ssSetSink(config);
}
   

Multisink with fully configurable console display

The outcome of the logging process is easy to configure with ssSetSink():

{
    // Always thread-safe, asynchronous storage, synchronous console display and live notifications
    sslog::Sink config;
    config.pathPattern      = "logdir";
    config.storageLevel     = sslog::Level::trace;
    config.consoleLevel     = sslog::Level::debug;
    config.detailsLevel     = sslog::Level::off;
    config.liveNotifLevel   = sslog::Level::error;
    config.liveNotifCbk     = myEmergencyLogProcessCallback;
    config.consoleFormatter = "[%L] [%Y-%m-%dT%H:%M:%S.%f%z] [%c] [thread %t] %v%Q";
    ssSetSink(config);

    ssTrace("myApp/main", "Message stored on disk but not displayed on console.");
}

Also, the log path may contain some date formatters:

{
    // Date formatters can be used on the log path
    sslog::Sink config;
    config.path = "logdir-%y%m%d-%HH%M_%S"; // Ex: logdir-250619-08H15_54
    ssSetSink(config);
}

See here for details on the logger configuration API.

   

Rotating files

No logger is complete without manipulation on the log files splitting and counting:

The ssSetSink() API is still the one to use:

{
    sslog::Sink config;
    config.path = "logdir";
    config.fileMaxBytes = 5 * 1024*1024; // Rotate after reaching 5 MB
    config.fileMaxDurationSec = 600;     // Or rotate after 10 mn
    config.fileMaxQty = 10;              // Keep the last 10 files
    config.fileMaxFileAgeSec = 3600;     // Keep only files more recent than 1 hour
    ssSetSink(config);
}
   

Selection of groups of log at compile-time

Some logs in some conditions may not be desirable, whatever their level.

Logs can be grouped and activated at compile time only when needed, with zero-cost when disabled.
Each logging API has a “group” variant, easily identified by the ssg prefix and the group name as first parameter:

// To define somewhere in the current file, a header, in the build system...
#define SSLOG_GROUP_VERY_LOW_LEVEL 0    # One group of logs (disabled)
#define SSLOG_GROUP_BINARY_DUMPS 1      # Another group of logs (enabled)

// When a group name is used, the prefix SSLOG_GROUP_ is not present

ssgError(VERY_LOW_LEVEL, "myApp/main", "First error with parameters: %d %-8s %f %08lx", 14, "stream", 3.14, 1234567890UL);

std::vector<uint8_t> buffer{0xDE, 0xAD, 0xBE, 0xEF};
ssgDebugBuffer(BINARY_DUMPS, "myApp/archive/dump", buffer.data(), buffer.size(),
             "Dump of received packet number=%d from client:%s_id", 314, "George");

See here for details.

   

FAQ

How can I further reduce the log storage size with zstd?

Once the logging session is finished, use for instance:

zstd --rm -r -5 sslogDb

The role of the options are:

If built with the zstd-dev package installed on the system, reader tools transparently read compressed log files.

I do not see the logged stracktrace when a crash occurs... Logging the stack trace requires these 2 points to be valid:

   

Dependencies

The repository does not require any external dependencies.

Details on internal dependencies are:

Optional dependencies:

   

License

Released under the MIT license.

  

@@ Logging configuration

  

@@ Logging reference API

  

@@ sslogread Python module

  

@@ sslogread C++ API

  

@@ sscat

formatted by Markdeep 1.13