(Top)
1 Overview
1.1 Features
1.2 Install
1.3 Benchmarks
1.4 Examples
1.4.1 Basic logging with names and units
1.4.2 Details on demand
1.4.3 Query with “sscat” shell tool
1.4.4 Structured output in JSON with “sscat” shell tool
1.4.5 Query in Python
1.4.6 Query in C++
1.4.7 Binary buffer logging
1.4.8 Log notification callback
1.4.9 Multisink with fully configurable console display
1.4.10 Rotating files
1.4.11 Selection of groups of log at compile-time
1.5 FAQ
1.6 Dependencies
1.7 License
2 Logging configuration
3 Logging reference API
4 sslogread Python module
5 sslogread C++ API
6 sscat
Speedy Structured C++17 logging library
To achieve 30 million logs per second, string information is pre-processed at compile-time so that practically only arguments are stored at run-time.
Furthermore, compact storage is obtained by writing duplicate strings only once, in binary form and with generalized delta-encoding.
Costly formatting is deferred to logs exploitation phase, enabling dynamic filtering, queries and transforms:
Logs act as a tiny database.
The flexible query interface is available in shell, python or C++, and is suitable for
In including just one header file.
sscat
: cat
-like tool with filters and transforms
libsslogread
: C++ library for reading logs
sslogread
module: wrapper for libsslogread
Copy and include the single header sslog.h
.
Then start logging!
The easiest way to install the python reader module is:
python3 -m pip install sslogread
Raw results from the internal benchmark (run test/run_suite.py -s performance
):
==============================================================================
| sslog - Logging runtime 0 param | 26.0 ns or 38.46 Mlog/s |
| sslog - Logging runtime 1 param int | 28.3 ns or 35.39 Mlog/s |
| sslog - Logging runtime 1 param string | 33.2 ns or 30.11 Mlog/s |
| sslog - Logging runtime 4 params int | 33.9 ns or 29.51 Mlog/s |
| sslog - Logging runtime skipped log | 0.1 ns or 13543.67 Mlog/s |
| | |
| sslog - compactness - full text | 100% (1e6 logs / 151.9 MB) |
| sslog - compactness - sslog vs text | 11.9 % |
| sslog - compactness - sslog+zstd -5 vs text | 3.0 % |
| | |
| sslog - Header include | 0.703 s |
| sslog - Header include - SSLOG_DISABLE=1 | 0.243 s |
| | |
| sslog - Library code size (-O2) | 45208 bytes |
| sslog - Log code size 0 param (-O2) | 60 bytes/log |
| sslog - Log code size 1 param int (-O2) | 80 bytes/log |
| sslog - Log code size 1 param string (-O2) | 72 bytes/log |
| sslog - Log code size 4 params int (-O2) | 112 bytes/log |
| | |
| sslog - Log compilation speed (-O0) | 6578 log/s |
| sslog - Log compilation speed (-O2) | 914 log/s |
==============================================================================
Highlights:
zstd -5
)
Raw results from the internal benchmark (run test/run_suite.py -s performance_spdlog
with spdlog-dev
installed):
==============================================================================
| spdlog - Logging runtime 0 param | 344.3 ns or 2.90 Mlog/s |
| spdlog - Logging runtime 1 param int | 349.0 ns or 2.87 Mlog/s |
| spdlog - Logging runtime 1 param string | 350.1 ns or 2.86 Mlog/s |
| spdlog - Logging runtime 4 params int | 410.6 ns or 2.44 Mlog/s |
| spdlog - Logging runtime skipped log | 2.5 ns or 392.63 Mlog/s |
| | |
| spdlog - Header include | 3.035 s |
| | |
| spdlog - Library code size (-O2) | 241840 bytes (+ shared libs) |
| spdlog - Log code size 0 param (-O2) | 20 bytes/log |
| spdlog - Log code size 1 param int (-O2) | 56 bytes/log |
| spdlog - Log code size 1 param string (-O2) | 56 bytes/log |
| spdlog - Log code size 4 params int (-O2) | 144 bytes/log |
| | |
| spdlog - Log compilation speed (-O0) | 11008 log/s |
| spdlog - Log compilation speed (-O2) | 2447 log/s |
==============================================================================
Highlights:
sslog
mainly due to the online formatting
spdlog
version 1.12 (from Ubuntu 24.04.2 LTS), compiled version
sslog
:
basic_file_sink_mt
block
overflow policy, no console
#include "sslog.h"
int main()
{
ssInfo("myApp/main", "First message on console with sslog!");
ssError("myApp/main", "First error with parameters: %d %-8s %f %08lx",
14, "stream", 3.14, 1234567890UL);
// The optional text before and after an argument is interpreted as name and unit
// The category (first string) aims typically user-specific classification
ssDebug("/animal/pet/dog", "surname=%s weight=%4.1f_kg", "mirza", 12.856);
ssTrace("/animal/pet/cat", "surname=%s weight=%4.1f_kg", "misty", 5.5);
}
See here for details on the logging API.
A unique feature is the logging of details upon request.
In the example below:
info
are always logged
trace
are logged only around the moment when “details are requested” #include "sslog.h"
int main()
{
// Configure the level of the details and the file split rules
sslog::Sink config;
config.storageLevel = sslog::Level::info; // Standard logging at "info" level
config.detailsLevel = sslog::Level::trace; // On request, enables details up to 'trace' (for storage)
config.fileMaxBytes = 10000; // Granularity of 10 KB
ssSetSink(config);
for (int iter = 0; iter < 100000; ++iter)
{
ssInfo("Always logged and stored");
ssDebug("Logged and stored only around the iteration 50000. Current iteration is %05d", iter);
// Request details up to 'trace' before and after this particular moment
// The duration of the surrounding context capture is configurable
if (iter == 50000) ssRequestForDetails();
}
}
Display all the logs (no filtering) in text:
sscat <log dir>
Select the logs on a category matching *engine*
but not on a thread matching worker*
sscat <log dir> -nt "worker*" -c "*engine*"
Select the logs on category transaction
and which possess an argument named id
which has a value higher than 356 and lower or equal to 1000
sscat <log dir> -c "transaction" -a "id>356" -a "id<=1000"
See here for details.
The option -j
switches to the JSON output:
sscat <log dir> -j
Display all the logs (no filtering) in text:
import sslogread
session = sslogread.load("/path/to/my/log_folder")
result = session.query()
for log in result:
print("%d) %s" % (log['timestampUtcNs'], log['format'])) # Simple display
Select the logs on a category matching *engine*
but not on a thread matching worker*
result = session.query( { 'no_thread': "worker*", 'category':'*engine*' } )
Select the logs on category transaction
and which possess an argument named id
which has a value higher than 356 and lower or equal to 1000
result = session.query( { 'category': "transaction", 'arguments': ["id>356", "id<=1000"] } )
See here for details on the Python reader module.
Display all the logs (no filtering) in text:
std::string errorMessage;
sslogread::LogSession session;
if(!session.init("/path/to/my/log/folder", errorMessage)) {
fprintf(stderr, "Error: %s\n", errorMessage.c_str());
exit(1)
}
if(!session->query({}, [session](const sslogread::LogStruct& log) {
// Format the string with arguments (=custom vsnprintf with our argument list, see below)
char filledFormat[1024];
sslogread::vsnprintfLog(filledFormat, sizeof(filledFormat), session->getIndexedString(log.formatIdx), log.args, session);
// Some simple display on console (there are better ways)
printf("[timestampUtcNs=%lu thread=%s category=%s buffer=%s] %s\n",
log.timestampUtcNs, session->getIndexedString(log.threadIdx), session->getIndexedString(log.categoryIdx),
log.buffer.empty()? "No" : "Yes", filledFormat);
},
errorMessage)
{
fprinf(stderr, "Error: %s\n", errorMessage.c_str());
}
Select the logs on a category matching *engine*
but not on a thread matching worker*
Rule rule;
rule.category = "*engine*";
rule.noThread = "worker*";
std::string errorMessage;
if(!session.query({rule}, [](const sslogread::LogStruct& log) { /* ... */ }, errorMessage)
{
fprinf(stderr, "Error: %s\n", errorMessage.c_str());
}
Select the logs on category transaction
and which possess an argument named id
which has a value higher than 356 and lower or equal to 1000
Rule rule;
rule.category = "transaction";
rule.arguments.push_back("id>356"); // Note: comparison works also with strings (alphanumerically)
rule.arguments.push_back("id<=1000");
/* ... */
See here for details on the C++ reader API.
The printf
interface provides built-in format checks,is wildly known and is quite expressive.
In this cotext, inserting support of custom structure cannot be easily supported.
An exception is made for the valuable case of generic binary buffers (dumps, images, packets, CAN messages...).
The API is similar to sslog
standard logging with the suffix “Buffer” and with additional arguments: buffer pointer and size before the format string.
#include "sslog.h"
#include <vector>
int main()
{
ssSetConsoleLevel(sslog::Level::trace);
std::vector<uint8_t> buffer{0xDE, 0xAD, 0xBE, 0xEF};
ssDebugBuffer("myApp/archive", buffer.data(), buffer.size(),
"Standard log with a binary buffer attached. Its sweet name is %s, id:%d",
"Francis", 314);
}
See here for details.
A custom log handler, or some special processing on high level logs?
Simply register a callback and set the start level.
#include "sslog.h"
int main()
{
sslog::Sink config;
config.liveNotifLevel = sslog::Level::critical;
config.liveNotifCbk = [] (uint64_t timestampUtcNs, uint32_t level, const char* threadName, const char* category,
const char* logContent, const uint8_t* binaryBuffer, uint32_t binaryBufferSize)
{
sendEmergencyMail("support@world.com", timestampUtcNs, category, logContent);
};
ssSetSink(config);
}
The outcome of the logging process is easy to configure with ssSetSink()
:
info
)
off
)
{
// Always thread-safe, asynchronous storage, synchronous console display and live notifications
sslog::Sink config;
config.pathPattern = "logdir";
config.storageLevel = sslog::Level::trace;
config.consoleLevel = sslog::Level::debug;
config.detailsLevel = sslog::Level::off;
config.liveNotifLevel = sslog::Level::error;
config.liveNotifCbk = myEmergencyLogProcessCallback;
config.consoleFormatter = "[%L] [%Y-%m-%dT%H:%M:%S.%f%z] [%c] [thread %t] %v%Q";
ssSetSink(config);
ssTrace("myApp/main", "Message stored on disk but not displayed on console.");
}
Also, the log path may contain some date formatters:
{
// Date formatters can be used on the log path
sslog::Sink config;
config.path = "logdir-%y%m%d-%HH%M_%S"; // Ex: logdir-250619-08H15_54
ssSetSink(config);
}
See here for details on the logger configuration API.
No logger is complete without manipulation on the log files splitting and counting:
The ssSetSink()
API is still the one to use:
{
sslog::Sink config;
config.path = "logdir";
config.fileMaxBytes = 5 * 1024*1024; // Rotate after reaching 5 MB
config.fileMaxDurationSec = 600; // Or rotate after 10 mn
config.fileMaxQty = 10; // Keep the last 10 files
config.fileMaxFileAgeSec = 3600; // Keep only files more recent than 1 hour
ssSetSink(config);
}
Some logs in some conditions may not be desirable, whatever their level.
Logs can be grouped and activated at compile time only when needed, with zero-cost when disabled.
Each logging API has a “group” variant, easily identified by the ssg
prefix and the group name as first parameter:
// To define somewhere in the current file, a header, in the build system...
#define SSLOG_GROUP_VERY_LOW_LEVEL 0 # One group of logs (disabled)
#define SSLOG_GROUP_BINARY_DUMPS 1 # Another group of logs (enabled)
// When a group name is used, the prefix SSLOG_GROUP_ is not present
ssgError(VERY_LOW_LEVEL, "myApp/main", "First error with parameters: %d %-8s %f %08lx", 14, "stream", 3.14, 1234567890UL);
std::vector<uint8_t> buffer{0xDE, 0xAD, 0xBE, 0xEF};
ssgDebugBuffer(BINARY_DUMPS, "myApp/archive/dump", buffer.data(), buffer.size(),
"Dump of received packet number=%d from client:%s_id", 314, "George");
See here for details.
zstd
? Once the logging session is finished, use for instance:
zstd --rm -r -5 sslogDb
The role of the options are:
-r
(recursive) ensures that each file in the folder are processed
--rm
removes the original files, once the compression is successful (safe)
-5
is the compression level, trade-off between speed and final size. It is adjusted from -1
(very fast) to -19
(high compression).
If built with the zstd-dev
package installed on the system, reader tools transparently read compressed log files.
libunwind-dev
and libdw-dev
on Linux)
The repository does not require any external dependencies.
Details on internal dependencies are:
sslog
uses only C++17 standard libraries.
sslogread
library internally uses a modified version of the vsnprintf code from stb library (Public domain)
Optional dependencies:
libdw
and libunwind
libraries are installed and the compilation flag SSLOG_STACKTRACE=1
is positioned, the stacktrace is logged when a crash occurs
libzstd
library is installed, sslogread
library reads and decompresses any .zst log files in a transparent manner.Released under the MIT license.