# Compression Support

CFGPack supports decompression of stored config blobs using LZ4 or heatshrink algorithms. This is useful when config blobs are stored compressed in flash to save space. Compression must be done externally (e.g., at build time or on the host); only decompression is supported on the device.

> **Note:** The `cfgpack_pagein_lz4()` and `cfgpack_pagein_heatshrink()` functions decompress **and then load config values** via `cfgpack_pagein_buf()`. They expect the decompressed result to be a config blob (produced by `cfgpack_pageout()`, with CRC-32C trailer). For compressed **schema** blobs (produced by `cfgpack-schema-pack`), call `LZ4_decompress_safe()` or heatshrink directly, then use `cfgpack_schema_parse_msgpack()`. See [`examples/fleet_gateway/`](../examples/fleet_gateway/) for the complete pattern.

Both LZ4 and heatshrink are enabled by default. To disable for minimal embedded builds, edit the `CFLAGS` in the Makefile to remove `-DCFGPACK_LZ4` and/or `-DCFGPACK_HEATSHRINK`.

## API

```c
#include "cfgpack/decompress.h"

/* Decompress LZ4 data and load into context.
 * decompressed_size must be known (stored alongside compressed data).
 * scratch/scratch_cap: caller-provided buffer for decompressed output. */
cfgpack_err_t cfgpack_pagein_lz4(cfgpack_ctx_t *ctx, const uint8_t *data, size_t len,
                                  size_t decompressed_size,
                                  uint8_t *scratch, size_t scratch_cap);

/* Decompress heatshrink data and load into context.
 * Encoder must use window=8, lookahead=4 to match decoder config.
 * scratch/scratch_cap: caller-provided buffer for decompressed output. */
cfgpack_err_t cfgpack_pagein_heatshrink(cfgpack_ctx_t *ctx, const uint8_t *data, size_t len,
                                         uint8_t *scratch, size_t scratch_cap);
```

## Usage Example

```c
// LZ4 example - decompressed size must be stored with the compressed blob
uint8_t compressed[2048];
uint8_t scratch[4096];  // caller-provided decompression buffer
size_t compressed_len = read_from_flash(compressed);
size_t decompressed_size = read_size_header();  // stored alongside blob

cfgpack_err_t err = cfgpack_pagein_lz4(&ctx, compressed, compressed_len, decompressed_size,
                                        scratch, sizeof(scratch));
if (err != CFGPACK_OK) {
    // Handle decompression or decode error
}

// Heatshrink example - no size header needed (streaming)
err = cfgpack_pagein_heatshrink(&ctx, compressed, compressed_len, scratch, sizeof(scratch));
```

## Implementation Notes

- **Caller-provided buffer**: Both decompression functions accept a `scratch` / `scratch_cap` parameter for the decompressed output. The caller controls the maximum decompressed size.
- **LZ4 path**: Fully reentrant — no static state.
- **Heatshrink path**: Uses a static decoder instance and is NOT thread-safe. The LZ4 path has no such limitation.
- **Heatshrink parameters**: The decoder is configured with window=8 bits (256 bytes) and lookahead=4 bits (16 bytes). The encoder must use matching parameters.
- **Vendored sources**: LZ4 and heatshrink source files are vendored in `third_party/` to avoid external dependencies.

## Compression Workflow

A typical workflow for storing compressed config:

1. **At build time or on host**: Serialize config with `cfgpack_pageout()`, compress with LZ4 or heatshrink, store compressed blob + size metadata in flash image
2. **On device boot**: Read compressed blob from flash, decompress with `cfgpack_pagein_lz4()` or `cfgpack_pagein_heatshrink()`

## Compression Tool

The `cfgpack-compress` CLI tool compresses files for build-time or host-side use:

```bash
make tools
./build/out/cfgpack-compress <algorithm> <input> <output>
```

Where `<algorithm>` is either `lz4` or `heatshrink`.

### Output Formats

- **LZ4**: The output file contains a 4-byte little-endian original (uncompressed) size header followed by the LZ4-compressed data:
  ```
  [0..3]  4-byte little-endian original size
  [4..N]  LZ4-compressed data
  ```
  This header is required because LZ4 decompression needs to know the output buffer size. The `examples/fleet_gateway/` example demonstrates reading this format at runtime. The `examples/flash_config/` example demonstrates the same pipeline with LittleFS flash storage.

- **Heatshrink**: The output file contains raw compressed data only (no header). Heatshrink is a streaming decoder and does not require the original size up front.

### Examples

```bash
# Compress a serialized config blob
./build/out/cfgpack-compress lz4 config.bin config.lz4
./build/out/cfgpack-compress heatshrink config.bin config.hs
```

## Schema Pack Tool

The `cfgpack-schema-pack` CLI tool converts `.map` or JSON schema files to compact MessagePack binary for over-the-wire delivery to devices:

```bash
make tools
./build/out/cfgpack-schema-pack <input> <output>
```

The input format is auto-detected: files ending in `.json` are parsed as JSON, all others as `.map` format.

Example:
```bash
./build/out/cfgpack-schema-pack schema.map schema.msgpack
./build/out/cfgpack-schema-pack schema.json schema.msgpack
```

The output binary can be parsed on-device with `cfgpack_schema_measure_msgpack()` and `cfgpack_schema_parse_msgpack()`. It can also be further compressed with `cfgpack-compress` for additional size savings on constrained links.

## Schema Validate Tool

The `cfgpack-schema-validate` CLI tool validates schema files in any supported format, with optional decompression:

```bash
make tools
./build/out/cfgpack-schema-validate [--lz4|--heatshrink] [--format map|json|msgpack] <input>
```

The input format is auto-detected by extension: `.json` for JSON, `.msgpack`/`.bin` for MessagePack binary, all others as `.map`. Use `--format` to override.

When `--lz4` or `--heatshrink` is specified, the file is decompressed first. Compressed schemas are always MessagePack binary underneath.

### Examples

```bash
# Validate a .map schema
./build/out/cfgpack-schema-validate schema.map

# Validate a JSON schema
./build/out/cfgpack-schema-validate schema.json

# Validate a compressed MessagePack schema
./build/out/cfgpack-schema-validate --lz4 schema.msgpack.lz4
./build/out/cfgpack-schema-validate --heatshrink schema.msgpack.hs

# Force format detection (useful when extension doesn't match)
./build/out/cfgpack-schema-validate --format msgpack schema.bin
```

On success, prints schema metadata and entry type summary:
```
Valid: "vehicle" v3 (12 entries)
  Types: 4 u8, 2 u16, 1 u32, 3 str, 2 fstr
```

On failure, prints the error with line number (for text formats) and exits with code 3:
```
Invalid: line 5: duplicate index 3
```

### Exit Codes

| Code | Meaning |
|------|---------|
| 0 | Valid schema |
| 1 | Usage error |
| 2 | File I/O error |
| 3 | Validation error |

## Third-Party Libraries

LZ4, heatshrink, and LittleFS sources are vendored in `third_party/` for self-contained builds:

### LZ4

```
third_party/lz4/
    lz4.h, lz4.c              # BSD-2-Clause license
```

### Heatshrink

```
third_party/heatshrink/
    heatshrink_config.h       # window=8, lookahead=4
    heatshrink_decoder.h/c    # Used by library
    heatshrink_encoder.h/c    # Used by compression tool and tests
                              # ISC license
```

### LittleFS

```
third_party/littlefs/
    lfs.h, lfs.c              # Core filesystem implementation
    lfs_util.h, lfs_util.c    # Utility functions
    LICENSE.md                 # BSD-3-Clause license
```

LittleFS is a little fail-safe filesystem designed for microcontrollers. Unlike LZ4 and heatshrink, it is not compiled into the core library — it is linked directly by applications that use the LittleFS wrappers (`-DCFGPACK_LITTLEFS`). See [LittleFS Storage Wrappers](littlefs.md) for details.