Compression Support

CFGPack supports decompression of stored config blobs using LZ4 or heatshrink algorithms. This is useful when config blobs are stored compressed in flash to save space. Compression must be done externally (e.g., at build time or on the host); only decompression is supported on the device.

Note: The cfgpack_pagein_lz4() and cfgpack_pagein_heatshrink() functions decompress and then load config values via cfgpack_pagein_buf(). They expect the decompressed result to be a config blob (produced by cfgpack_pageout(), with CRC-32C trailer). For compressed schema blobs (produced by cfgpack-schema-pack), call LZ4_decompress_safe() or heatshrink directly, then use cfgpack_schema_parse_msgpack(). See examples/fleet_gateway/ for the complete pattern.

Both LZ4 and heatshrink are enabled by default. To disable for minimal embedded builds, edit the CFLAGS in the Makefile to remove -DCFGPACK_LZ4 and/or -DCFGPACK_HEATSHRINK.

API

#include "cfgpack/decompress.h"

/* Decompress LZ4 data and load into context.
 * decompressed_size must be known (stored alongside compressed data).
 * scratch/scratch_cap: caller-provided buffer for decompressed output. */
cfgpack_err_t cfgpack_pagein_lz4(cfgpack_ctx_t *ctx, const uint8_t *data, size_t len,
                                  size_t decompressed_size,
                                  uint8_t *scratch, size_t scratch_cap);

/* Decompress heatshrink data and load into context.
 * Encoder must use window=8, lookahead=4 to match decoder config.
 * scratch/scratch_cap: caller-provided buffer for decompressed output. */
cfgpack_err_t cfgpack_pagein_heatshrink(cfgpack_ctx_t *ctx, const uint8_t *data, size_t len,
                                         uint8_t *scratch, size_t scratch_cap);

Usage Example

// LZ4 example - decompressed size must be stored with the compressed blob
uint8_t compressed[2048];
uint8_t scratch[4096];  // caller-provided decompression buffer
size_t compressed_len = read_from_flash(compressed);
size_t decompressed_size = read_size_header();  // stored alongside blob

cfgpack_err_t err = cfgpack_pagein_lz4(&ctx, compressed, compressed_len, decompressed_size,
                                        scratch, sizeof(scratch));
if (err != CFGPACK_OK) {
    // Handle decompression or decode error
}

// Heatshrink example - no size header needed (streaming)
err = cfgpack_pagein_heatshrink(&ctx, compressed, compressed_len, scratch, sizeof(scratch));

Implementation Notes

  • Caller-provided buffer: Both decompression functions accept a scratch / scratch_cap parameter for the decompressed output. The caller controls the maximum decompressed size.

  • LZ4 path: Fully reentrant — no static state.

  • Heatshrink path: Uses a static decoder instance and is NOT thread-safe. The LZ4 path has no such limitation.

  • Heatshrink parameters: The decoder is configured with window=8 bits (256 bytes) and lookahead=4 bits (16 bytes). The encoder must use matching parameters.

  • Vendored sources: LZ4 and heatshrink source files are vendored in third_party/ to avoid external dependencies.

Compression Workflow

A typical workflow for storing compressed config:

  1. At build time or on host: Serialize config with cfgpack_pageout(), compress with LZ4 or heatshrink, store compressed blob + size metadata in flash image

  2. On device boot: Read compressed blob from flash, decompress with cfgpack_pagein_lz4() or cfgpack_pagein_heatshrink()

Compression Tool

The cfgpack-compress CLI tool compresses files for build-time or host-side use:

make tools
./build/out/cfgpack-compress <algorithm> <input> <output>

Where <algorithm> is either lz4 or heatshrink.

Output Formats

  • LZ4: The output file contains a 4-byte little-endian original (uncompressed) size header followed by the LZ4-compressed data:

    [0..3]  4-byte little-endian original size
[4..N]  LZ4-compressed data

    This header is required because LZ4 decompression needs to know the output buffer size. The examples/fleet_gateway/ example demonstrates reading this format at runtime. The examples/flash_config/ example demonstrates the same pipeline with LittleFS flash storage.

  • Heatshrink: The output file contains raw compressed data only (no header). Heatshrink is a streaming decoder and does not require the original size up front.

Examples

# Compress a serialized config blob
./build/out/cfgpack-compress lz4 config.bin config.lz4
./build/out/cfgpack-compress heatshrink config.bin config.hs

Schema Pack Tool

The cfgpack-schema-pack CLI tool converts .map or JSON schema files to compact MessagePack binary for over-the-wire delivery to devices:

make tools
./build/out/cfgpack-schema-pack <input> <output>

The input format is auto-detected: files ending in .json are parsed as JSON, all others as .map format.

Example:

./build/out/cfgpack-schema-pack schema.map schema.msgpack
./build/out/cfgpack-schema-pack schema.json schema.msgpack

The output binary can be parsed on-device with cfgpack_schema_measure_msgpack() and cfgpack_schema_parse_msgpack(). It can also be further compressed with cfgpack-compress for additional size savings on constrained links.

Schema Validate Tool

The cfgpack-schema-validate CLI tool validates schema files in any supported format, with optional decompression:

make tools
./build/out/cfgpack-schema-validate [--lz4|--heatshrink] [--format map|json|msgpack] <input>

The input format is auto-detected by extension: .json for JSON, .msgpack/.bin for MessagePack binary, all others as .map. Use --format to override.

When --lz4 or --heatshrink is specified, the file is decompressed first. Compressed schemas are always MessagePack binary underneath.

Examples

# Validate a .map schema
./build/out/cfgpack-schema-validate schema.map

# Validate a JSON schema
./build/out/cfgpack-schema-validate schema.json

# Validate a compressed MessagePack schema
./build/out/cfgpack-schema-validate --lz4 schema.msgpack.lz4
./build/out/cfgpack-schema-validate --heatshrink schema.msgpack.hs

# Force format detection (useful when extension doesn't match)
./build/out/cfgpack-schema-validate --format msgpack schema.bin

On success, prints schema metadata and entry type summary:

Valid: "vehicle" v3 (12 entries)
  Types: 4 u8, 2 u16, 1 u32, 3 str, 2 fstr

On failure, prints the error with line number (for text formats) and exits with code 3:

Invalid: line 5: duplicate index 3

Exit Codes

Code

Meaning

0

Valid schema

1

Usage error

2

File I/O error

3

Validation error

Third-Party Libraries

LZ4, heatshrink, and LittleFS sources are vendored in third_party/ for self-contained builds:

LZ4

third_party/lz4/
    lz4.h, lz4.c              # BSD-2-Clause license

Heatshrink

third_party/heatshrink/
    heatshrink_config.h       # window=8, lookahead=4
    heatshrink_decoder.h/c    # Used by library
    heatshrink_encoder.h/c    # Used by compression tool and tests
                              # ISC license

LittleFS

third_party/littlefs/
    lfs.h, lfs.c              # Core filesystem implementation
    lfs_util.h, lfs_util.c    # Utility functions
    LICENSE.md                 # BSD-3-Clause license

LittleFS is a little fail-safe filesystem designed for microcontrollers. Unlike LZ4 and heatshrink, it is not compiled into the core library — it is linked directly by applications that use the LittleFS wrappers (-DCFGPACK_LITTLEFS). See LittleFS Storage Wrappers for details.