At-least-once, honestly
A source offset commits only after every record derived from it is durably written or intentionally dropped — held across rebalances, graceful shutdown, and crashes. The watermark stalls rather than ever advancing past unacknowledged data.
Sources never block
Backpressure pauses lanes and keeps polling, so a source thread never parks in a channel send. Sink slowness throttles throughput without ever triggering a consumer-group eviction.
No hidden costs on the hot path
Operator chains are fully monomorphized — one virtual call per batch, not per record — and every metric handle is pre-registered at build time. Measured at ~9 ns/record with zero per-record allocations.
Batteries-included connectors
Kafka in, Avro decoding, sharded and replicated ClickHouse out — each a small, stable trait you can swap for your own. First-class Prometheus metrics, /healthz//readyz probes, and drain-on-SIGTERM come standard.
A taste
Operators are stateful closures composed into one monomorphized loop; YAML carries the tuning and connector configuration, never the topology.
let chains = move |_thread| {
chain_owned::<Order, _>(avro.clone())
.with_metrics("orders", "main")
.try_map(validate, ErrorPolicy::Skip)
.map(enrich)
.sink(ClickHouseEncoder::new(), KeyHashRouter,
ChunkConfig::default(), queues.clone(), budget.clone())
.build()
};
PipelineRuntime::new(config, kafka_source, chains, sink, budget).run()?;