Module sink 

Sink abstraction: pipeline threads encode, shard workers batch and write.

The division of labour (see docs/DESIGN.md § Sink):

• Pipeline threads run the sink’s RowEncoder inside the chain’s terminal stage, accumulating encoded rows into small EncodedChunk frames per shard and try_sending them into bounded per-shard queues (never blocking — a full queue surfaces as backpressure).
• Shard workers (tokio tasks) merge chunks from all pipeline threads into full-size batches, seal on max_rows / max_bytes / linger, and dispatch up to max_inflight concurrent ShardWriter::write_batch calls rotating across healthy replicas. Merging at the worker keeps batches large regardless of the pipeline thread count.

A connector implements RowEncoder (CPU half) and ShardWriter (I/O half); the framework owns everything between them.

Structs

BatchConfig

Batch sealing thresholds for one shard worker. A batch seals as soon as any threshold trips; since chunks arrive whole, a sealed batch may overshoot max_rows/max_bytes by at most one chunk.

BreakerConfig

Per-replica circuit breaker thresholds.

ChunkSendError

A rejected chunk, handed back so the terminal stage can park it and report Blocked upstream.

DrainReport

What a full-pool drain accomplished.

EncodedChunk

A small frame of encoded rows produced on a pipeline thread, the unit shipped over the per-shard queues. Wire frames are concatenable — either the format is headerless (RowBinary rows appended back-to-back) or each frame is one complete, self-describing block (ClickHouse Native), and a concatenation of complete blocks is itself a legal insert stream — so workers accumulate chunks without re-encoding.

InflightConfig

In-flight write limits for one shard worker.

KeyHashRouter

Default router: key hash modulo shards, falling back to the source partition for keyless records (keeps a partition’s keyless records together and the distribution stable).

RetryConfig

Retry policy for batch writes. Retries rotate across healthy replicas; the sealed batch and its deduplication token are reused unchanged.

SealedBatch

A batch sealed by a shard worker, ready to write. Frames concatenate to the full wire payload (a stream of one or more self-describing blocks for block formats like ClickHouse Native).

ShardQueues

Sending side of every shard queue, shared by pipeline threads.

SinkParts

The decomposed sink. Construct with SinkParts::new and refine with the with_* methods (the struct is #[non_exhaustive]; fields may be added without breaking implementors).

SinkPool

Shard workers plus the handles to probe and drain them.

SinkPoolConfig

Complete sink worker-pool configuration.

Traits

RowEncoder

The CPU half of a sink connector: encodes one record into the sink’s wire format. Runs on pinned pipeline threads inside the chain’s terminal stage; must not perform I/O. Family-generic and dyn-compatible, like Deserializer.

ShardRouter

Routes records to shards. Pure and cheap — called per record on pipeline threads.

ShardWriter

The I/O half of a sink connector: writes one sealed batch to one replica endpoint. Returning Ok is the durable-ack point — only then may the framework resolve the batch’s acknowledgements.

SinkBundle

A sink ready for assembly: the writer, the shard/replica topology, pool tuning, and optional metadata (metric labels, readiness probe).

Functions

endpoint_probe

Build a SinkProbeFn that probes every replica of every shard in shard_endpoints (indexed [shard][replica]) via ShardWriter::probe — the readiness loop SinkParts::with_probe expects. Back writer with an independent probe client set, never the insert clients (see SinkParts::probe).

shard_queues

Build the queues: one bounded channel per shard. Returns the shared sender handle and the per-shard receivers for the workers.

Type Aliases

SinkDrainFn

Boxed sink drain hook: budget in, report out. Produced by sink assemblies (wrapping SinkPool::drain), consumed once at shutdown by the pipeline runtime.

SinkProbeFn

Boxed, repeatable sink connectivity probe (readiness). The runtime probes at startup and then periodically, driving the sinks-connected half of /readyz.