Pipes and Filters

Problem

If the whole transformation lives in one monolithic function, the stages are tangled together and none of them can be tested in isolation; a bug in the OCR step is only reachable by running the entire pipeline end to end. If the team writes a bespoke pipeline each time, every project reinvents the plumbing for connecting one stage to the next and the stages cannot be shared across pipelines. Both extremes block the reuse and isolated testing the team wants.

Solution

Decompose the transformation into small filters with single responsibilities. Connect them via typed pipes (function call, queue, stream). Each filter is testable in isolation. Filters can be reused across pipelines.

When to use

• A transformation can be decomposed into small filters with single responsibilities.
• Filters benefit from being individually testable and reusable across pipelines.
• Typed pipes (call, queue, stream) connect filters cleanly.