Pipe
Pipe — Workflow Orchestration
- *rea:*Data Platform
- *ath:*
platform/pipe - *ind:*Workflow / pipeline orchestration engine
Role in the stack
Pipe is the workflow and pipeline orchestration engine. It is where longrunning, multistep, asynchronous processes are defined and executed: ETL jobs, data migrations, CI pipelines, batch processing, webhook fan-outs. Consumers describe a pipeline as a DAG of steps; Pipe schedules, retries, and observes them.
Primary couplings
| Consumer | Relationship |
|---|---|
platform/kdb |
State storage (pipeline runs, step outputs) |
data/mq |
Job queueing between steps |
platform/ci |
CI pipelines use Pipe as the execution engine |
| `observe |