DistilCodeAll techniques

Technique 06 / 14 · Noise reduction

JSON compressor

Turn large API responses and data files into a compact view without dropping keys or anomalies.

Interactive model

See the context change.

“Why did order 1019 fail?” against a 48-order API response.

Compressed4 rows + marker

GET /orders

  1. 01

    First row: 1001 paid

  2. 02

    Error row: 1019 failed, card_declined

  3. 03

    Outlier: 1042 refunded, $4,980

  4. 04

    Last row: 1048 paid

  5. 05

    44 similar rows dropped (viewraw)

Conceptual demonstration. Counts illustrate the flow, not measured production savings.

How it works

Less context. Same thread of work.

API responses and data dumps repeat the same keys on every row. DistilCode’s JSON compressor, a port of headroom’s SmartCrusher, first tries a lossless column table for arrays of objects. When that is still too large it samples rows, always keeping the first and last, error and outlier rows, and rows that match your latest request.

Object keys are never dropped, and a marker stands in for the rows left out. Config files, manifests and lockfiles the agent may edit by hand are left exact. The original output is kept, so the viewraw tool can always return it.

In practice

The operating loop

  1. 01

    Lossless first

    Arrays of objects become a column table before any row is sampled.

  2. 02

    Keep the anchors

    First/last rows, errors, outliers and rows matching the request survive sampling.

  3. 03

    Always recoverable

    Editable JSON stays exact, and viewraw returns any compressed original.