Struct Pipeline 

pub struct Pipeline { /* private fields */ }

The pipeline builder — see the module docs for the full picture.

Non-generic, nameable, and storable: the source type enters only at the terminal into_runtime/run call.

Implementations

impl Pipeline

pub fn from_path(path: &Path) -> Result<Pipeline, BuildError>

Load configuration from a YAML file and initialize the process; see from_config.

pub fn from_config(config: PipelineConfig) -> Result<Pipeline, BuildError>

Initialize the process from an already-loaded configuration:

1. Telemetry — telemetry::init(Json, "info"). Idempotent: to customize the format or filter, call telemetry::init yourself first (the binaries-init convention).
2. Metrics exporter — installed from the config’s metrics section before you can construct any handle, so every handle built while holding the Pipeline is live. When a foreign recorder already owns the process, the pipeline continues against it with a warning.
3. The I/O runtime — pipeline.io_threads workers, thread name etl-io. Connectors that need a handle before run (schema fetchers, async pre-flight validation) use io_handle/block_on.

Errors

BuildError::AsyncContext when called from inside an async runtime — build pipelines from a plain thread, usually main.

pub fn config(&self) -> &PipelineConfig

The loaded configuration — connector sections (config().source, .deserializer, .sink) still belong to the caller’s connector factories.

pub fn metrics(&self) -> &MetricsHandle

The installed exporter’s handle (rendering, upkeep).

pub fn budget(&self) -> &Arc<InflightBudget>

The shared in-flight byte budget.

pub fn io_handle(&self) -> Handle

A handle to the I/O runtime, for connector edge work that must start before the chain exists (schema-registry fetchers, …). Valid until run returns.

pub fn block_on<F>(&self, future: F) -> <F as Future>::Output

where F: Future,

Run a future on the I/O runtime, blocking this thread — for async pre-flight steps such as schema validation.

pub fn sink<B>(self, bundle: B) -> Result<Pipeline, BuildError>

where B: SinkBundle,

Install the sink with default SinkOptions; see sink_with.

pub fn sink_with<B>( self, bundle: B, options: SinkOptions, ) -> Result<Pipeline, BuildError>

where B: SinkBundle,

Install the sink: builds the per-shard chunk queues, registers the per-shard metrics (E2E basis from the config), spawns the SinkPool workers on the I/O runtime, and wires the drain and readiness probe.

Errors

BuildError::SinkAlreadySet on a second call; BuildError::Sink for an empty or ragged topology, label shapes that do not match it, or a zero queue capacity.

pub fn chains<F>(self, factory: F) -> Pipeline

where F: FnMut(ChainCtx) -> Box<dyn RunnableChain> + Send + 'static,

Install the chain factory, called once per pipeline thread with that thread’s ChainCtx. Composition inside the closure is fully monomorphized (chain_owned and friends); the returned Box<dyn RunnableChain> is the same single per-batch erasure boundary as always.

pub fn runtime_options(self, options: RuntimeOptions) -> Pipeline

Override the runtime options (signal handling, loop timings).

pub fn into_runtime<S>( self, source: S, ) -> Result<PipelineRuntime<S>, BuildError>

where S: Source + 'static,

Finish assembly into a PipelineRuntime — for callers that need shutdown_handle before a spawned run (tests, embedded pipelines). The I/O runtime moves into it and is shut down when run returns.

Errors

BuildError::MissingSink / BuildError::MissingChains when a step was skipped.

pub fn run<S>(self, source: S) -> Result<ExitReport, PipelineError>

where S: Source + 'static,

into_runtime + PipelineRuntime::run: run the pipeline to completion, blocking until a shutdown signal drains it or a fatal error stops it.

Trait Implementations

impl Debug for Pipeline

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter.