S3 backfill baseline

What does a bounded backfill cost with the network taken out of the loop? This rig stages NDJSON objects on a local filesystem store and runs the real thing end to end: the startup listing, per-lane fetchers, streaming decompression and framing on the pipeline threads, checkpoint commits to the manifest, and the self-terminating drain. The sink is a null writer, so the number is the source's decode-and-frame ceiling — a real S3 backfill is then bounded by network and request latency, not by this machinery.

Because the job is bounded, the honest metric is total wall time for the whole job, not a steady-state window: every figure below includes listing and the final synchronous commit.

Backfill throughput by codec

Higher is better

Records through the full pipeline per second of end-to-end wall time (64 objects × 20k records × 256 B). Higher is better.

Data table

Variant	Value	95% CI	n
zstd	7,420,781.003227327 records/s	—	—
gzip	7,163,459.083287104 records/s	—	—
uncompressed	3,902,223.8905042317 records/s	—	—

Apple M5 Max · commit ed1fe3da72 · 2026-07-14

Decoded bytes per second

Higher is better

Decompressed NDJSON bytes framed per second. Compressed objects come out ahead here because far fewer stored bytes cross the store — the decompressors are cheaper than the extra reads they save. Lower stored volume is also what a real S3 backfill wants: fewer bytes over the network.

Data table

Variant	Value	95% CI	n
zstd	1,899.7199368261956 MB/s	—	—
gzip	1,833.8455253214986 MB/s	—	—
uncompressed	998.9693159690833 MB/s	—	—

Apple M5 Max · commit ed1fe3da72 · 2026-07-14

Reading the numbers

• No GO/NO-GO gate. This is a new capability, not an alternative implementation competing against an incumbent; the baseline exists so regressions in the source's hot path (framing, decompression, the chunk channel) show up as a falling bar.
• Compressed beats plain on a local store. gzip and zstd read far fewer stored bytes (the ratio is the payload's compression ratio — this rig's synthetic records compress by two orders of magnitude; real-world JSON compresses less) and pay a modest decompress cost on the pipeline threads. Against real S3 the effect is stronger — stored bytes are network bytes.
• The rig: benchmarks/src/bin/s3_backfill.rs (OBJECTS, RECORDS_PER_OBJECT, PAYLOAD, LANES, THREADS, CODECS env knobs). Results append to benchmarks/results/s3-backfill.jsonl via RESULTS.