Chunk sizing and queue capacity

The chain terminal seals a chunk once its encoded buffer reaches ChunkConfig::target_bytes (default 64 KiB) and hands it to a per-shard bounded queue (SinkOptions::queue_capacity, default 8). Neither knob bounds insert size — shard workers merge chunks into full batches. What they govern is handoff rate and how long a chunk waits before a worker sees it.

This page exists because both defaults look small, and the obvious instinct is to raise them. The measurements say: raise the chunk target only when you have a specific symptom, and leave queue capacity alone.

Why the earlier thread-scaling curves were misleading

Framework overhead reports a synthetic ceiling that peaks at 2 pipeline threads, and Sink saturation peaks at 4. Neither is a framework scaling limit: both rigs swept THREADS while holding egress width fixed — pipeline_synthetic hardcoded 2 I/O workers, ch_sink_saturation pinned shards and io_threads across its sweep. Each curve peaked at roughly its own fixed egress width.

The pipeline_scaling rig sweeps threads with SHARDS and IO_THREADS scaled alongside (EGRESS=scaled) and the in-flight budget derived per arm from the sizing rule. Under those conditions the framework keeps scaling, and the chunk target decides how far.

Framework ceiling vs pipeline threads, by chunk target

Higher is better

Generator source, null writer, no broker and no server — the framework alone. Egress width scaled with thread count; in-flight budget derived per arm. Medians of 3.

1 MiB64 KiB (default)

Data table

Group	Series	Value	95% CI	n
1	1 MiB	50,596,763.11810256 records/s	—	3
1	64 KiB (default)	41,000,817.87046864 records/s	—	3
2	1 MiB	100,594,219.5774915 records/s	—	3
2	64 KiB (default)	74,616,405.33689356 records/s	—	3
4	1 MiB	178,671,815.05701533 records/s	—	3
4	64 KiB (default)	115,793,386.27237774 records/s	—	3
6	1 MiB	244,578,169.56498495 records/s	—	3
6	64 KiB (default)	118,191,307.34747325 records/s	—	3
8	1 MiB	301,920,478.855918 records/s	—	3
8	64 KiB (default)	111,132,771.77075565 records/s	—	3

Apple M5 Max · commit a0a05abfa2 · 2026-07-18

At the default 64 KiB the curve flattens and then falls back, while backpressure pause events climb steeply. At 1 MiB the pauses stay at zero across the whole sweep and scaling efficiency at 8 threads is roughly double.

Backpressure pause events over the same window

Lower is better

Lower is better. The sink here is a null writer that completes instantly — every one of these pauses is the handoff rate, not sink slowness.

1 MiB64 KiB (default)

Data table

Group	Series	Value	95% CI	n
1	1 MiB	0 events	—	3
1	64 KiB (default)	2 events	—	3
2	1 MiB	0 events	—	3
2	64 KiB (default)	4 events	—	3
4	1 MiB	0 events	—	3
4	64 KiB (default)	16 events	—	3
6	1 MiB	0 events	—	3
6	64 KiB (default)	94 events	—	3
8	1 MiB	0 events	—	3
8	64 KiB (default)	167 events	—	3

Apple M5 Max · commit a0a05abfa2 · 2026-07-18

That second chart is the load-bearing one, because the sink is a null writer that completes instantly. There is no sink slowness available to signal. The pauses are caused purely by chunks arriving faster than the shard workers can service them — the queue-full trigger is rate-driven, not only capacity-driven.

The same lever against a real sink

The synthetic rig runs an order of magnitude above any real sink, so the question is whether the lever survives contact with one. It does, though substantially attenuated — and it is not free.

Throughput vs chunk target (ClickHouse, ENGINE=Null)

Higher is better

8 pipeline threads, queue capacity 256, in-flight budget pinned at 8 GiB so the budget is identical across arms. Medians of 3.

1 MiB (highlighted)other arms

Data table

Variant	Value	95% CI	n
1 MiB	4,855,650.854132424 rows/s	—	3
512 KiB	4,323,788.15454432 rows/s	—	3
128 KiB	3,879,192.2190376557 rows/s	—	3
256 KiB	3,635,393.6059428793 rows/s	—	3
64 KiB (default)	3,542,854.417787189 rows/s	—	3

SYNTHETIC FIXTURE · Apple M5 Max · commit a0a05abfa2 · 2026-07-18

Flush p99 vs chunk target — the cost side

Lower is better

Same arms. Lower is better: throughput and flush latency rise together.

1 MiB (highlighted)other arms

Data table

Variant	Value	95% CI	n
64 KiB (default)	2.3949999999999996 s	—	3
128 KiB	2.4166666666666665 s	—	3
256 KiB	3.5166666666666417 s	—	3
512 KiB	3.5583333333333416 s	—	3
1 MiB	4.402083333333332 s	—	3

SYNTHETIC FIXTURE · Apple M5 Max · commit a0a05abfa2 · 2026-07-18

There is no knee. Throughput and flush latency climb in near lockstep across the whole range, so no chunk target dominates the others. The shipped 64 KiB default sits deliberately at the low-latency end. Raise it when you have the specific symptom — backpressure pauses while the sink itself is idle — and accept the latency in exchange.

Pin the budget when comparing chunk sizes

The sizing rule makes the in-flight budget a function of chunk.target_bytes, so a naive A/B that lets the budget float changes two variables at once. Doing that here inflated the apparent gain by roughly a third. Every arm above pins MAX_INFLIGHT_MB to the same value.

Queue capacity: the small default is the right one

Throughput vs queue capacity

Higher is better

8 pipeline threads at the default 64 KiB chunk target, budget pinned. Medians of 3.

8 (default) (highlighted)other arms

Data table

Variant	Value	95% CI	n
1024	3,924,085.7608097848 rows/s	—	3
256	3,542,854.417787189 rows/s	—	3
64	2,795,115.7681006156 rows/s	—	3
8 (default)	2,694,910.8278902927 rows/s	—	3

SYNTHETIC FIXTURE · Apple M5 Max · commit a0a05abfa2 · 2026-07-18

Flush p99 vs queue capacity

Lower is better

Same arms. The default holds the best p99 by an order of magnitude.

8 (default) (highlighted)other arms

Data table

Variant	Value	95% CI	n
8 (default)	0.2478238341968912 s	—	3
64	1.952499999999997 s	—	3
256	2.3949999999999996 s	—	3
1024	2.4934375000000006 s	—	3

SYNTHETIC FIXTURE · Apple M5 Max · commit a0a05abfa2 · 2026-07-18

This one is not a smooth trade, and the shape matters more than the endpoints. Moving off the default to the next step up pays roughly an order of magnitude of flush p99 for a few percent of throughput — a clearly bad trade, because the queue bounds the latency of whatever sits in it and the first step away from a shallow queue is where almost all of that latency appears. Past that point the curve inverts: the deepest arm buys substantially more throughput for comparatively little further latency.

So the practical guidance is narrower than "keep it small":

• The default is latency-optimal by a wide margin, and that is what Tuning means by signal, not buffer.
• If you leave it, leave it decisively. An intermediate depth pays most of the latency cost for very little of the throughput gain.
• The throughput on the table is not negligible — this is a real choice for a throughput-dominant pipeline, not a knob to leave alone on principle.

Reading the pause counter

etl_backpressure_pause_events_total is not on its own a misconfiguration signal. In the queue-capacity arms above, the configuration with the best latency also recorded the most pauses. Pauses mean the source was throttled; whether that is healthy depends on what was throttling it.

Symptom	Likely cause	Lever
Pauses high, sink flush p99 low, sink not saturated	chunk handoff rate	raise chunk.target_bytes
Pauses high, sink flush p99 high	sink genuinely behind	more shards / inflight.max_per_shard / a faster sink
Pauses high, etl_backpressure_inflight_bytes near the cap	budget undersized	apply the sizing rule

etl_queue_full_events_total is blind in these rigs

That counter is registered by ShardQueues::attach_metrics, which is only reached through the pipeline builder. Every benchmark rig assembles the runtime directly, so the series is structurally always zero there and cannot be used as a validity check. The pause counter and flush histogram are the usable signals. Pipelines built through the normal builder API do record it.

Caveats

Single 18-core machine; ClickHouse runs in a container on the same cores, so absolute rates are not a cluster result — the within-run A/B is what these arms establish. pipeline_scaling arms stay within threads + io_threads + 1 <= cores; the rig marks any arm that exceeds it OVERSUBSCRIBED in its note, and none of the published arms do. The synthetic rig's payload is low-entropy filler, which flatters compression-free paths. Reproduce with the invocations in Methodology.