CFGPack Infrastructure
This document describes in detail all the tools, systems, and methods used to build, test, format, fuzz, document, and maintain the CFGPack C library.
Table of Contents
Build System
The project uses a single GNU Makefile at the repository root. There are no CMake, Meson, or Autotools files. The build is intentionally minimal and self-contained, with no external package manager dependencies beyond a C compiler and standard POSIX tools.
Build Directories
All build artifacts are placed under build/:
Path |
Contents |
|---|---|
|
Embedded-mode |
|
Hosted-mode |
|
Scratch files written by tests and tool smoke tests |
|
Final outputs: static library, test binaries, tool binaries |
|
Per-TU JSON fragments for |
|
Generated documentation (Doxygen XML, Sphinx HTML) |
Makefile Targets
Target |
Description |
|---|---|
|
Build |
|
Build |
|
Build all test binaries into |
|
Build |
|
Run |
|
Build every example under |
|
Build all libFuzzer harnesses (delegated to |
|
Generate Sphinx + Doxygen documentation |
|
Auto-format all source with clang-format |
|
Dry-run format check (fails on diff – used by CI/hooks) |
|
Generate |
|
Build at |
|
Build at |
|
Rebuild tests with ASan+UBSan and run the full test suite |
|
Rebuild with LLVM coverage, run tests, and generate report |
|
Remove all build artifacts, compile_commands.json, fuzz corpora |
|
Remove generated docs and the Python venv |
|
List all targets with descriptions |
Dependency Tracking
The Makefile uses Clang’s -MMD -MP flags to emit .d dependency files alongside each .o. These are included at the bottom of the Makefile with -include $(DEPS), giving automatic incremental rebuilds when any header changes.
Compiler and Toolchain
Setting |
Value |
|---|---|
Compiler |
|
Archiver |
|
C standard |
C99 ( |
Warning flags |
|
Optimization |
|
Formatter |
|
The Makefile uses @ prefixes to silence command echo, printing shorter custom status lines instead (e.g., CC src/core.c, AR build/out/libcfgpack.a, LD build/out/basic).
Conditional Compilation
The library has two build modes controlled by preprocessor defines:
CFGPACK_EMBEDDED (default)
No
<stdio.h>dependency in the core library.cfgpack_print()/cfgpack_print_all()become silent no-ops.Float formatting uses a minimal internal implementation (no
snprintf).CFGPACK_PRINTF(...)expands to((void)0).
CFGPACK_HOSTED
Enabled by compiling with -DCFGPACK_HOSTED. Provides full printf/snprintf support. The Makefile defines a separate CFLAGS_HOSTED variable and uses it for:
All test object files (
tests/*.c)src/io_file.c(file I/O wrapper requiring<stdio.h>)All CLI tools
This is enforced with pattern-specific rules in the Makefile:
# Test objects need CFLAGS_HOSTED for printf support
$(OBJ)/tests/%.o: tests/%.c
$(CC) $(CPPFLAGS) $(CFLAGS_HOSTED) ...
# io_file.c needs CFLAGS_HOSTED for stdio
$(OBJ)/src/io_file.o: src/io_file.c
$(CC) $(CPPFLAGS) $(CFLAGS_HOSTED) ...
Optional Compression Support
Two additional feature flags gate decompression library support:
-DCFGPACK_LZ4– enables LZ4 decompression (on by default in CFLAGS)-DCFGPACK_HEATSHRINK– enables Heatshrink decompression (on by default in CFLAGS)
These are also passed to Doxygen as PREDEFINED macros so that conditional API surfaces appear in documentation.
Optional LittleFS Storage Wrappers
-DCFGPACK_LITTLEFS– enables LittleFS storage wrappers (cfgpack_pageout_lfs,cfgpack_pagein_lfs)
This flag is on by default in CFLAGS, and src/io_littlefs.c along with the vendored LittleFS sources (third_party/littlefs/lfs.c, third_party/littlefs/lfs_util.c) are compiled into the core library. This flag is also passed to Doxygen as a PREDEFINED macro.
Compile-Time Limits
Defined in include/cfgpack/config.h and overridable before including cfgpack headers:
Macro |
Default |
Purpose |
|---|---|---|
|
128 |
Max schema entries; determines inline presence bitmap size |
|
32 |
Max nesting depth for msgpack skip (32 levels = 128 bytes stack) |
Static Library Production
The default make target produces build/out/libcfgpack.a – a static archive of the core library. The core sources are:
src/core.c
src/crc32.c
src/decompress.c
src/io.c
src/io_littlefs.c
src/msgpack.c
src/schema_parser.c
src/tokens.c
src/wbuf.c
third_party/lz4/lz4.c
third_party/heatshrink/heatshrink_decoder.c
third_party/littlefs/lfs.c
third_party/littlefs/lfs_util.c
Notably, src/io_file.c is excluded from the core library because it depends on <stdio.h>. It is compiled separately with hosted flags and linked into tests and tools as needed. This preserves the embedded-friendly, zero-stdio core.
The archiver creates the library with ar rcs.
Test Infrastructure
Framework
The project uses a custom minimal test framework defined in tests/test.h (116 lines). There are no external test framework dependencies.
Core types and macros:
Macro/Type |
Purpose |
|---|---|
|
Enum: |
|
Declares a test function: |
|
Assert; on failure prints file:line and the expression, then returns |
|
Assert with success logging |
|
Print dimmed verbose output |
|
Print a cyan section header |
|
Hex-dump a buffer (up to 32 bytes) |
|
Print a |
Shared implementation in tests/test.c:
test_case_result(name, result)– prints colored PASS/FAIL and returns the result.
Test Binary Structure
Each test file has its own main() that runs its test cases and returns a combined pass/fail status:
int main(void) {
test_result_t overall = TEST_OK;
overall |= (test_case_result("basic", test_basic_case()) != TEST_OK);
overall |= (test_case_result("pageout_small_buffer",
test_pageout_small_buffer()) != TEST_OK);
/* ... */
if (overall == TEST_OK) {
printf("ALL PASS\n");
} else {
printf("SOME FAIL\n");
}
return (overall);
}
Each test binary links against: the core static library, io_file.o, the encoder dependency objects, and the shared test.o.
Test Binaries
17 test files producing 16 test binaries (test.c is shared infrastructure, not a standalone binary):
Binary |
Source |
Area |
|---|---|---|
|
|
Core set/get/pageout/pagein, defaults, typed convenience functions |
|
|
Edge cases in core API |
|
|
Typed convenience wrappers, file I/O, init bounds, presence API |
|
|
LZ4 and heatshrink decompression |
|
|
I/O edge cases |
|
|
LittleFS I/O wrappers (RAM-backed block device) |
|
|
JSON parser edge cases |
|
|
JSON remapping functionality |
|
|
Schema measure (pre-parse sizing) |
|
|
MessagePack encode/decode |
|
|
MessagePack decoder edge cases (wide format codes, skip depth) |
|
|
MessagePack schema handling |
|
|
NULL pointer and bounds validation |
|
|
Schema parser |
|
|
Parser boundary conditions |
|
|
Runtime behavior |
Test Runner Script
scripts/run-tests.sh orchestrates test execution:
Changes to the repository root (tests open fixtures relative to it) and creates
build/tmp/.Iterates over a hardcoded list of test binary names.
Runs each binary under a 120-second
timeoutwhen one is available, capturing stdout/stderr and exit code.Strips colour codes and counts lines anchored at
PASS/FAIL, so each case is counted exactly once (the indentedCHECKdiagnostics and theALL PASS/SOME FAILsummaries are excluded).Prints a per-binary summary with color-coded pass/fail counts.
Writes full output to
build/test.log.Prints a total summary and exits non-zero if any test failed.
A missing binary or a timed-out binary is reported as a failure, not skipped.
Fuzz Testing
Architecture
Fuzz testing uses LLVM libFuzzer with AddressSanitizer (ASan) and UndefinedBehaviorSanitizer (UBSan). It is driven by a dedicated sub-Makefile at tests/fuzz/Makefile, invoked from the root Makefile via make fuzz.
Platform Detection
The fuzz Makefile includes automatic platform detection for macOS:
Checks if the selected compiler’s resource directory contains
libclang_rt.fuzzer_osx.a.If not (typically Apple Clang, which does not ship libFuzzer), it falls back to Homebrew LLVM at the path returned by
brew --prefix llvm.On macOS, it also adds linker flags (
-Land-Wl,-rpath) to resolve LLVM’slibc++.
On Linux, system Clang with libFuzzer typically works out of the box.
Fuzz Flags
-fsanitize=fuzzer,address,undefined -fno-omit-frame-pointer -g
Library sources are compiled directly into each fuzz binary (rather than linking the static archive) because sanitizers require instrumentation at both compile and link time.
Fuzz Harnesses
6 fuzz targets, each implementing LLVMFuzzerTestOneInput:
Target |
Corpus Dir |
What It Fuzzes |
|---|---|---|
|
|
|
|
|
JSON schema parser |
|
|
MessagePack schema parser |
|
|
MessagePack parser with custom mutator |
|
|
Configuration page-in (deserialization) |
|
|
LZ4 and heatshrink decompression wrappers with fuzzer-chosen scratch sizes |
|
|
Raw MessagePack decoding |
Seed Corpus Generator
tests/fuzz/gen_seeds.c is a standalone hosted program that generates valid binary seed files for each corpus directory. It produces well-formed inputs (valid JSON schemas, valid MessagePack blobs, valid page-in buffers, etc.) that give the fuzzer a meaningful starting point. The generator runs automatically as a dependency of make fuzz via the gen-seeds target.
Corpus directories are gitignored and regenerated at build time.
Running Fuzz Tests
scripts/run-fuzz.sh runs all fuzz targets sequentially:
Default duration: 60 seconds per target.
Configurable via argument:
scripts/run-fuzz.sh 300for 300 seconds.Use
scripts/run-fuzz.sh 0for indefinite fuzzing.Max input length: 4096 bytes.
Prints the last 20 lines of libFuzzer output per target.
Reports overall pass/fail status.
Code Formatting
clang-format Configuration
All formatting is enforced by .clang-format at the repository root, based on the LLVM style with project-specific overrides:
Setting |
Value |
Notes |
|---|---|---|
|
4 |
4-space indent |
|
Never |
Spaces only |
|
80 |
80-column hard limit |
|
Attach |
K&R brace style |
|
true |
Braces inserted for single-statement blocks |
|
Right |
|
|
false |
Function parameters one-per-line when wrapping |
|
true |
Arguments may be bin-packed |
|
true |
Compact case labels |
|
BeforeHash |
Preprocessor |
|
false |
Preserves section banner comments |
|
CaseSensitive |
Auto-sorts includes |
Include Ordering
Includes are automatically regrouped (IncludeBlocks: Regroup) into three priority groups separated by blank lines:
Priority 1:
cfgpack/library headers (#include "cfgpack/...")Priority 2: Other local headers (
#include "tokens.h", etc.)Priority 3: System headers (
#include <string.h>, etc.)
Formatting Targets
make format– auto-formats all.cand.hfiles undersrc/,include/,tests/,examples/, andtools/(excludingthird_party/).make format-check– dry-run with--Werror; exits non-zero if any file would change. Used by the pre-commit hook.
The set of files to format is discovered dynamically via find.
Git Hooks
Pre-Commit Hook
Located at scripts/pre-commit and symlinked into .git/hooks/pre-commit by the setup script.
The hook runs two stages on every commit:
Format check –
make format-check. Fails with a message directing the developer to runmake format.Build and test –
make testsfollowed byscripts/run-tests.sh. Fails with the test summary if any test fails.
The hook extracts and displays the TOTAL: summary line (with ANSI colors stripped) on success or failure.
It can be bypassed with git commit --no-verify when needed.
Hook Installation
The symlink is created by scripts/setup.sh:
ln -sf ../../scripts/pre-commit "$ROOT_DIR/.git/hooks/pre-commit"
Using a symlink (rather than a copy) means the hook automatically stays in sync with the version in scripts/.
Documentation Generation
Documentation uses a Doxygen + Sphinx + Breathe pipeline:
Pipeline
Doxygen parses the public headers in
include/cfgpack/and generates XML output tobuild/docs/doxygen/xml/.Breathe (a Sphinx extension) reads the Doxygen XML and makes it available as Sphinx directives.
Sphinx renders the final HTML documentation to
build/docs/html/.
Doxygen Configuration
The Doxyfile at the repository root is configured for a C library:
Setting |
Value |
|---|---|
|
YES |
|
|
|
|
|
YES (for Breathe) |
|
NO (Sphinx handles HTML) |
|
YES |
|
YES |
|
YES |
|
|
All graph generation |
NO (no Graphviz dependency) |
Sphinx Configuration
docs/conf.py configures:
Extensions:
breathe(Doxygen bridge),myst_parser(Markdown support)Theme:
sphinx_rtd_theme(Read the Docs theme)Source formats: both
.rst(reStructuredText) and.md(Markdown via MyST)MyST extensions:
colon_fence,deflist
Python Dependencies
Managed via docs/requirements.txt and installed into an isolated venv at docs/.venv/:
sphinx>=7.0
breathe>=4.35
sphinx-rtd-theme>=2.0
myst-parser>=2.0
The make docs target automatically creates the venv and installs dependencies if they don’t exist.
Documentation Source Files
docs/
conf.py # Sphinx configuration
index.rst # Main table of contents
getting-started.rst # Getting started guide
api-reference.md # API reference (Markdown)
compression.md # Compression documentation
fuzz-testing.md # Fuzz testing guide
littlefs.md # LittleFS integration
stack-analysis.md # Stack usage analysis
versioning.md # Schema versioning
infrastructure.md # This file
CLI Tools
Three CLI tools are built with make tools:
cfgpack-compress
Source: tools/cfgpack-compress.c
Compresses files with LZ4 or Heatshrink for use with the library’s decompression support.
Usage: cfgpack-compress <algorithm> <input> <output>
Algorithms: lz4, heatshrink
LZ4 output: 4-byte little-endian original size + raw compressed data.
Heatshrink output: raw compressed data (window=8, lookahead=4).
Links against vendored LZ4 and Heatshrink encoder sources directly.
Max input/output: 64 KB.
cfgpack-schema-pack
Source: tools/cfgpack-schema-pack.c
Converts .map or JSON schemas to MessagePack binary format for on-device loading.
Usage: cfgpack-schema-pack <input> <output>
Auto-detects input format by file extension (
.json= JSON, otherwise.map).Output is raw MessagePack binary for
cfgpack_schema_parse_msgpack().Links against the core library and
io_file.o.
cfgpack-schema-validate
Source: tools/cfgpack-schema-validate.c
Validates schema files in any supported format, with optional LZ4 or heatshrink decompression.
Usage: cfgpack-schema-validate [--lz4|--heatshrink] [--format map|json|msgpack] <input>
Auto-detects input format by extension (
.json= JSON,.msgpack/.bin= MessagePack, otherwise.map).--formatflag overrides extension-based detection.--lz4/--heatshrinkdecompresses before validating (assumes msgpack binary underneath).Runs two-phase validation: measure (structure check) then full parse (catches duplicates).
On success, prints schema name, version, entry count, and type summary to stdout.
On failure, prints error message with line number (for text formats) to stderr.
Links against the core library and
io_file.o.
All tools use exit code conventions: 0 = success, 1 = usage error, 2 = I/O error, 3 = processing error.
Third-Party Dependencies
All third-party code is vendored in third_party/ with no submodules or package managers:
LZ4
third_party/lz4/
lz4.c
lz4.h
LICENSE
LZ4 block-level compression/decompression. The decoder (lz4.c) is compiled into the core library when CFGPACK_LZ4 is defined. The compressor is only needed by the cfgpack-compress tool.
Heatshrink
third_party/heatshrink/
heatshrink_decoder.c
heatshrink_decoder.h
heatshrink_encoder.c
heatshrink_encoder.h
heatshrink_common.h
heatshrink_config.h
LICENSE
Heatshrink is an LZSS-based compression library designed for embedded systems with very low memory overhead. The decoder is compiled into the core library when CFGPACK_HEATSHRINK is defined. The encoder is used by tests (to generate compressed test data) and the cfgpack-compress tool.
LittleFS
third_party/littlefs/
lfs.c
lfs.h
lfs_util.c
lfs_util.h
LICENSE.md
LittleFS is a little fail-safe filesystem designed for microcontrollers. The LittleFS sources are compiled directly into applications that use the cfgpack LittleFS wrappers (they are not part of the core libcfgpack.a).
All three libraries are included via -Ithird_party/lz4, -Ithird_party/heatshrink, and -Ithird_party/littlefs in CPPFLAGS.
Stack Analysis
The Makefile provides two targets for analyzing per-function stack usage, useful for verifying the library’s suitability for stack-constrained embedded environments:
make stack-usage-O0– Builds the core library at-O0(no optimization) with Clang’s-fstack-usageflag, then collects and sorts all.sufiles by stack size (descending).make stack-usage-Os– Same at-Os(size-optimized, the default optimization level).
Both targets do a clean first to ensure all TUs are rebuilt with the stack-usage flag. The output shows each function’s stack frame size, allowing developers to identify hot spots and verify that the library stays within embedded stack budgets.
Compile Commands Database
make compile_commands generates a compile_commands.json at the repository root by concatenating per-TU JSON fragments produced by Clang’s -MJ flag during compilation. This provides IDE integration (clangd, ccls, etc.) with accurate flags for each translation unit.
The per-TU fragments are written to build/json/ during normal compilation and assembled with sed into a valid JSON array.
Project Setup
scripts/setup.sh is a one-time setup script for new clones:
Checks required tools:
clang,clang-format,make,ar– reports pass/fail for each.Checks optional tools: Looks for Homebrew LLVM on macOS (for fuzz testing), or verifies libFuzzer support on Linux.
Installs the pre-commit hook: Creates a symlink from
.git/hooks/pre-committoscripts/pre-commit.Verifies build and tests: Runs
make testsandscripts/run-tests.shto confirm the project builds and all tests pass.Prints a summary: Color-coded count of passed checks, warnings, and failures. Exits non-zero if any required check failed.
Directory Layout
cfgpack/
├── Makefile # Root build system
├── Doxyfile # Doxygen configuration
├── .clang-format # Code formatting rules
├── .gitignore # Git ignore rules
├── CLAUDE.md # AI assistant guidelines
├── include/cfgpack/ # Public headers
│ ├── cfgpack.h # Umbrella header
│ ├── api.h # Core API
│ ├── schema.h # Schema types
│ ├── value.h # Value types
│ ├── error.h # Error codes
│ ├── config.h # Build configuration / compile-time limits
│ ├── msgpack.h # MessagePack encode/decode
│ ├── io_file.h # File I/O (hosted only)
│ ├── io_littlefs.h # LittleFS I/O (optional, -DCFGPACK_LITTLEFS)
│ └── decompress.h # Decompression API
├── src/ # Library implementation
│ ├── core.c # Core get/set/init/pageout/pagein
│ ├── crc32.c # CRC-32C (Castagnoli) implementation
│ ├── crc32.h # CRC-32C internal header
│ ├── schema_parser.c # .map and JSON schema parsing
│ ├── msgpack.c # MessagePack codec
│ ├── decompress.c # LZ4/heatshrink decompression
│ ├── io.c # Buffer I/O
│ ├── io_file.c # File I/O (hosted, excluded from core lib)
│ ├── io_littlefs.c # LittleFS I/O (optional, excluded from core lib)
│ ├── tokens.c # Tokenizer for schema parsing
│ └── wbuf.c # Internal write buffer
├── tests/ # Test files
│ ├── test.h # Test framework header
│ ├── test.c # Shared test infrastructure
│ ├── basic.c ... runtime.c # 16 test binaries
│ ├── data/ # Test fixture files
│ └── fuzz/ # Fuzz testing
│ ├── Makefile # Fuzz build system
│ ├── gen_seeds.c # Seed corpus generator
│ ├── fuzz_parse_map.c # 7 fuzz harness sources
│ ├── fuzz_parse_json.c
│ ├── fuzz_parse_msgpack.c
│ ├── fuzz_parse_msgpack_mutator.c
│ ├── fuzz_pagein.c
│ ├── fuzz_decompress.c
│ └── fuzz_msgpack_decode.c
├── tools/ # CLI tools
│ ├── cfgpack-compress.c # LZ4/heatshrink compression tool
│ ├── cfgpack-schema-pack.c # Schema-to-msgpack converter
│ └── cfgpack-schema-validate.c # Schema validation tool
├── examples/ # Usage examples
│ ├── allocate-once/ # One-shot allocation pattern
│ ├── datalogger/ # Data logger example
│ ├── flash_config/ # LittleFS + LZ4 + schema migration
│ ├── fleet_gateway/ # Fleet gateway with schema versioning
│ ├── low_memory/ # Low-memory embedded example
│ └── sensor_hub/ # Sensor hub example
├── third_party/ # Vendored dependencies
│ ├── lz4/ # LZ4 compression library
│ ├── heatshrink/ # Heatshrink compression library
│ └── littlefs/ # LittleFS filesystem library
├── scripts/ # Build/test/setup scripts
│ ├── setup.sh # One-time project setup
│ ├── run-tests.sh # Test runner with summary
│ ├── run-coverage.sh # LLVM coverage report generator
│ ├── run-fuzz.sh # Fuzz test runner
│ ├── doc-check-diff.sh # Documentation drift checker
│ └── pre-commit # Git pre-commit hook
└── docs/ # Documentation sources
├── conf.py # Sphinx configuration
├── requirements.txt # Python dependencies for Sphinx
├── index.rst
├── getting-started.rst
├── api-reference.md
├── compression.md
├── littlefs.md
├── fuzz-testing.md
├── stack-analysis.md
├── versioning.md
└── infrastructure.md # This file