I · ReasoningEmerging★

Generate-and-Test Strategy

also known as Multi-Hypothesis with Constraint Verification, Hypothesize-then-Test

Generate multiple candidate solutions in parallel, then systematically test each against declared constraints rather than committing to the first plausible one — adapted from Langley & Simon's cognitive-science research on human expert problem-solving.

Context

The agent faces a problem with multiple plausible solutions and known constraints. Default LLM behavior is to commit to the first fluent answer (premature-closure). Expert humans, by contrast, generate alternatives and check each against constraints before committing.

Problem

Single-path generation commits prematurely to suboptimal solutions. Multi-path generation alone (e.g. tree-of-thoughts) explores but doesn't always systematically verify against declared constraints. The team needs the discipline of generation-then-verification as a unit.

Forces

  • Generating multiple hypotheses costs N× per attempt.
  • Constraint verification requires explicit constraint statement up front.
  • Some domains have hard constraints (math) and others soft (style); the test step must handle both.

Example

A trading-strategy agent given 'maximize Q4 returns under risk constraint X'. Naive: pick the first plausible strategy. Generate-and-Test: generate 5 candidate strategies, test each against historical data + risk constraint + drawdown limit. Three fail constraint X (one violates drawdown, one fails risk, one fails returns floor). Two pass; pick the higher-return one. Without the Test step, the agent would have shipped a constraint-violating strategy.

Diagram

Solution

Therefore:

Two-stage workflow. Generate: produce K candidates using multi-path or sampling. Test: for each candidate, verify against declared constraints (deterministic where possible, LLM-judge where soft). Pick the highest-passing candidate or escalate if none passes. Distinct from multi-path-plan-generator (which generates candidates without mandating verification). Pair with strategic-preparation-phase (which provides the constraint list), planner-executor-verifier, multi-path-plan-generator.

What this pattern forbids. No candidate is committed without passing the Test step; the constraint list is declared up front, not invented during generation.

And the patterns that stand alongside it, or against it —

  • complementsMulti-Path Plan Generator★★— Generate multiple candidate next-steps at each plan node enabling later selection — the planning generator pattern paired with tree-of-thoughts / LATS-style search.
  • complementsStrategic Preparation Phase★— Mandate an explicit problem-space representation step before the agent attempts solutions, mirroring how expert humans build a mental model of constraints and dependencies before solving.
  • complementsPlanner-Executor-Verifier (PEV)★— Triadic specialization where a planner produces the plan, an executor runs it, and a separate verifier checks each step's effects against the original goal.
  • complementsBest-of-N Sampling★— Sample N candidate outputs and select the highest-ranked by a reward model or scorer.
  • alternative-toPremature Closure✕— The LLM commits to a confident answer before processing all constraints, characteristic of constraint-heavy tasks where it fills in plausible answers fast and gets cross-constraint interactions wrong.
  • alternative-toContext Fragmentation✕— Anti-pattern: the LLM cannot hold multiple interconnected constraints in mind simultaneously the way human working memory can; it processes each constraint locally and loses the cross-constraint view.
  • complementsLarge Reasoning Model (LRM) Paradigm★— Route reasoning-heavy tasks to a reasoning-tuned model that trades inference time for deliberation, rather than to a fast LLM that exhibits premature-closure.
  • alternative-toSilent Hypotheses in Generated Code✕— Anti-pattern: model-written code rests on an unstated runtime premise that passing tests and code review never surface, so the hidden assumption travels into production and fails there.

Neighbourhood

Click any neighbour to follow the language. Scroll to zoom, drag to pan.

Used in frameworks

References

Provenance