Module backpressure 

Backpressure: the global in-flight byte budget and the watermark pause/resume controller with hysteresis.

Invariant (see docs/DESIGN.md § Backpressure): source threads never block on sends. When a try_send is rejected or the in-flight budget crosses its high watermark, the poll loop pauses its source lanes and keeps polling; it resumes only under hysteresis — usage back below the low watermark, downstream queues drained, and a minimum pause elapsed — so pause/resume cannot flap faster than once per min_pause.

Everything here is synchronous and tokio-free: pipeline threads call it on every poll iteration, and the InflightBudget atomics are modeled under loom. Run the loom suite with:

RUSTFLAGS="--cfg loom" cargo test -p etl-core --release backpressure::loom_tests

Poll-loop integration

use etl_core::backpressure::{
    BackpressureParams, InflightBudget, Transition, WatermarkController,
};
use std::sync::Arc;
use std::time::Duration;

let budget = Arc::new(InflightBudget::new());
let params = BackpressureParams::from_budget(
    256 * 1024 * 1024, // max in-flight bytes
    0.8,               // pause at 80%
    0.5,               // resume below 50%
    Duration::from_millis(500),
);
let mut controller = WatermarkController::new(params);

// Inside the poll loop:
// - when a try_send to a shard queue is rejected:
//     controller.on_send_rejected();
//     (stash the undeliverable record; NEVER block)
// - once per iteration:
let queues_below_low = true; // driver-provided: all shard queues < 50% full
match controller.tick(&budget, queues_below_low) {
    Some(Transition::Pause) => { /* source.pause(&owned_lanes) */ }
    Some(Transition::Resume) => { /* source.resume(&owned_lanes) */ }
    None => {}
}

Structs

BackpressureParams

Hysteresis parameters for one pipeline’s watermark controller.

InflightBudget

Global in-flight byte budget, shared by pipeline threads (which add on enqueue to sink queues) and sink workers (which subtract when a batch is acknowledged or abandoned).

MonotonicClock

Default Clock over Instant::now.

WatermarkController

Per-pipeline-thread pause/resume state machine with hysteresis.

Enums

Transition

A pause or resume decision for the poll loop to apply to its source lanes (and mirror into the backpressure metrics).

Traits

Clock

Time source for the controller, injectable so hysteresis is testable without sleeping.