Struct KafkaSource 

pub struct KafkaSource { /* private fields */ }

Kafka source: one consumer-group member per process, partitions split into per-lane queues polled by pipeline threads. Constructed from config (KafkaSource::new) or a pipeline component section (KafkaSource::from_component_config).

Implementations

impl KafkaSource

pub fn new(config: KafkaSourceConfig) -> Self

Create a source from validated configuration.

pub fn from_component_config( section: &ComponentConfig, ) -> Result<Self, ConfigError>

Create a source from the pipeline’s opaque source: { kafka: ... } section.

pub fn with_metrics(self, metrics: SourceMetrics) -> Self

Attach pre-registered source metrics (consumer lag, rebalances). Optional; without it the source only logs.

Trait Implementations

impl Debug for KafkaSource

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Drop for KafkaSource

Teardown: consumer close (inside BaseConsumer::drop) triggers a final revoke and then polls until the rebalance protocol completes — with the deferred-intent design, nothing would ever complete it and the drop would hang forever. Flip the context to inline-completion mode and settle any revocation that was surfaced but not yet acknowledged.

fn drop(&mut self)

Executes the destructor for this type.

fn pin_drop(self: Pin<&mut Self>)

🔬This is a nightly-only experimental API. (pin_ergonomics)

Execute the destructor for this type, but different to Drop::drop, it requires self to be pinned.

impl Source for KafkaSource

type Lane = KafkaLane

The lane type this source produces.

fn open(&mut self, ctx: SourceCtx) -> Result<(), SourceError>

Connect and prepare. Called once before any other method.

fn poll_events( &mut self, timeout: Duration, ) -> Result<SourceEvent<KafkaLane>, SourceError>

Service control-plane work (rebalance callbacks, statistics) and return the next event, waiting at most timeout. Must be called regularly regardless of backpressure state.

fn commit( &mut self, watermarks: &[(PartitionId, i64)], ) -> Result<(), SourceError>

Store per-partition committable positions (each is the offset one past the last acknowledged record). Positions are durable per the source’s own policy (e.g. interval auto-commit of stored offsets).

fn flush_commits(&mut self) -> Result<(), SourceError>

Synchronously flush stored positions (shutdown, revocation).

fn pause(&mut self, lanes: &[LaneId]) -> Result<(), SourceError>

Stop fetching for lanes (backpressure). Optional capability: sources that cannot pause rely on bounded-queue pushback alone.

fn resume(&mut self, lanes: &[LaneId]) -> Result<(), SourceError>

Resume fetching for lanes.