II · Planning & Control FlowEmerging★

MapReduce for Agents

also known as LLM×MapReduce, Divide-and-Conquer

Split an oversize task into independent chunks, process each in parallel, then aggregate.

This pattern helps complete certain larger patterns —

  • specialisesParallelization★★— Run independent LLM calls concurrently and combine results.
  • used-byGraphRAG★— Build an LLM-extracted entity-and-relation knowledge graph plus hierarchical community summaries, then answer global queries via map-reduce over those summaries.
  • used-byIteration Node★★— Express map-over-collection inside a visual workflow as an explicit Iteration node that runs a subgraph once per element of an input array, with bounded, deterministic, observable execution.
  • used-byQuery-Decomposition Agent★★— An agent whose explicit job is to split an incoming user query into smaller independent sub-queries that can be answered sequentially or in parallel, then merge results.

Context

A team needs to apply a language model to an input that is too large for a single call — twelve hundred pages of vendor contracts, a million-row table, hundreds of documents to summarise — or to a task that decomposes naturally into independent pieces (per row, per document, per section). Per-piece work is short; what is hard is the scale.

Problem

Stuffing the whole input into a long-context model still degrades quality past a certain point; quality drops in the middle of long documents and the model conflates entities across the input. Chunking the input and processing each chunk in isolation loses anything that depends on more than one chunk, such as cross-document deduplication or per-entity aggregation. Without a structured reduction step, conflicts between chunk answers go unresolved, and the team ends up either rerunning the whole thing in a giant call or hand-merging chunk outputs.

Forces

  • Naive chunking loses dependencies that span chunks.
  • Conflicts between chunk answers need a resolver.
  • Aggregation must not become its own context-window problem.

Example

A compliance team needs to extract every clause about data-residency from a 1200-page set of vendor contracts. A single long-context call drops clauses past page 400 and conflates two vendors. The team applies map-reduce: each contract is chunked, each chunk runs a clause-extraction prompt in parallel, and a reduce step aggregates per-vendor with a confidence-calibration prompt that resolves contradictions between chunks. Coverage rises and the run completes in twelve minutes instead of an hour-long sequential crawl.

Diagram

Solution

Therefore:

Map: split input into chunks; process each independently (per-chunk LLM call). Reduce: aggregate intermediate answers via a structured information protocol that surfaces dependencies, plus a confidence-calibration step to resolve conflicts.

What this pattern forbids. Each Map step sees only its chunk; cross-chunk reasoning is forbidden until the Reduce stage.

The smaller patterns that complete this one —

  • generalisesLLM Map-Reduce Isolation★— Process each untrusted document in its own sealed sub-agent and merge only structured outputs, so an injection in one document cannot steer the processing of others.

And the patterns that stand alongside it, or against it —

  • alternative-toSelf-Consistency★★— Sample the same question multiple times at non-zero temperature and aggregate by majority or judge to mitigate hallucination.
  • composes-withPipes and Filters★★— Compose stream-shaped processing as a chain of small filters connected by pipes.
  • alternative-toParallel Fan-Out / Gather★— Multiple independent agents execute in parallel on a partitioned task; a dedicated aggregator agent reconciles their results into a single output.
  • alternative-toScatter-Gather Plus Saga★— Distribute tasks across worker agents and aggregate results while maintaining distributed-transaction semantics via compensating actions on partial failure.

Neighbourhood

Click any neighbour to follow the language. Scroll to zoom, drag to pan.

References

Provenance