etl-rs User Guide

etl-rs is a high-performance, at-least-once ETL pipeline framework in Rust. Its flagship shape is Kafka in, Avro decoding, a chain of operators you write in Rust, and sharded ClickHouse out — but every piece behind that shape is a small, stable trait, so you can swap in your own sources, sinks, and deserializers. You define the operator graph in code (Flink / Java Streams style); YAML configures connectors and tuning, never the topology.

One process runs one pipeline. CPU work (polling, deserialization, the operator chain) runs on pinned threads with zero per-record allocations on the hot path; I/O (sink writes, schema fetches, the admin server) runs on a small shared tokio runtime. The design is Kubernetes-native: scrape-based Prometheus metrics, /healthz and /readyz probes, and drain-on-SIGTERM.

Guarantees at a glance

Guarantee	What it means
At-least-once delivery	A source offset commits only after every record derived from it was durably written to the sink (or intentionally dropped). This holds across consumer rebalances, graceful shutdown, and crashes. See Delivery guarantees.
Duplicates are possible	At-least-once means replays happen. Dedup tokens make same-batch retries idempotent, but crash replay re-batches with new boundaries — design target tables to tolerate duplicates.
Sources never block	Backpressure pauses lanes and keeps polling; a source thread never parks in a channel send, so sink slowness never triggers a consumer-group eviction. See Backpressure.
Errors are Skip or Fail	Record-level error policies are Skip (count and continue) or Fail (stop the pipeline) — never silent, always surfaced through metrics. See Error handling.
No hidden costs on the hot path	Operator chains are fully monomorphized (one virtual call per batch, not per record); metric handles are pre-registered at build time.

Not in scope for v1: exactly-once sinks, general DAG topologies, windowing, dead-letter queues, and config hot-reload. The rationale lives in docs/DESIGN.md — the source of truth when code and intuition disagree.

Where to start

• Never used etl-rs? Getting started has a five-minute in-memory pipeline and a full Kafka-to-ClickHouse tutorial.
• Want the mental model first? Read Concepts.
• Building or operating a real pipeline? Jump to the Guides and Deployment.

Guide map

1. Getting started

• Installation — the etl facade crate, feature flags, MSRV.
• Quickstart — a zero-infrastructure pipeline in five minutes.
• Your first pipeline — Kafka → Avro → operator chain → ClickHouse, end to end.

2. Concepts

• Architecture — the anatomy of a running pipeline process.
• Delivery guarantees — how at-least-once actually works, and its honest limits.
• Backpressure — flow control without blocking the source.
• Error handling — Skip, Fail, and the metrics that surface both.

3. Guides

• Assembling a pipeline
• Configuring pipelines
• Testing pipelines
• Schema validation
• Graceful shutdown
• Manual assembly

4. Connectors

• Kafka · ClickHouse · Avro · Memory (testing)

5. Deployment

• Docker · Monitoring · Tuning

6. Extending

• Custom sources · Custom sinks · Custom operators

7. Reference

• Configuration · Glossary

Related documentation

• docs/DESIGN.md — the architecture and every decision's rationale (canonical; this guide links to it rather than restating it).
• docs/METRICS.md — the full metric taxonomy and alerting starting points.
• API reference on docs.rs — this guide teaches concepts and contracts; signatures and per-item docs live in the rustdoc for the etl crate.