Streaming / chunked large payloads
A multi-gigabyte file should never be loaded into memory all at once. Stream it through the pipeline in chunks, so the platform processes data far larger than it could ever hold whole — and keep the heavy bytes off the bus entirely.
Streaming means each pipeline component reads, transforms and writes the payload in passes rather than materialising it in full. Memory use stays flat regardless of size, throughput stays predictable, and a huge transfer cannot exhaust resources. For routing and tracking, pair it with the claim check: the bus carries a small reference token while the bulk data streams from storage to destination.
The constraint streaming imposes is that not every operation can see the whole payload at once — a transform that needs global context (sorting the entire file, say) breaks the streaming model. Where you can work locally, per record or per chunk, streaming lets size stop being a limit.