I · ReasoningEmerging★

Generate-and-Test Strategy

also known as Multi-Hypothesis with Constraint Verification, Hypothesize-then-Test

Generate multiple candidate solutions in parallel, then systematically test each against declared constraints rather than committing to the first plausible one — adapted from Langley & Simon's cognitive-science research on human expert problem-solving.

Context

The agent faces a problem with multiple plausible solutions and known constraints. Default LLM behavior is to commit to the first fluent answer (premature-closure). Expert humans, by contrast, generate alternatives and check each against constraints before committing.

Problem

Single-path generation commits prematurely to suboptimal solutions. Multi-path generation alone (e.g. tree-of-thoughts) explores but doesn't always systematically verify against declared constraints. The team needs the discipline of generation-then-verification as a unit.

Forces

  • Generating multiple hypotheses costs N× per attempt.
  • Constraint verification requires explicit constraint statement up front.
  • Some domains have hard constraints (math) and others soft (style); the test step must handle both.

Example

A trading-strategy agent given 'maximize Q4 returns under risk constraint X'. Naive: pick the first plausible strategy. Generate-and-Test: generate 5 candidate strategies, test each against historical data + risk constraint + drawdown limit. Three fail constraint X (one violates drawdown, one fails risk, one fails returns floor). Two pass; pick the higher-return one. Without the Test step, the agent would have shipped a constraint-violating strategy.

Diagram

Solution

Therefore:

Two-stage workflow. Generate: produce K candidates using multi-path or sampling. Test: for each candidate, verify against declared constraints (deterministic where possible, LLM-judge where soft). Pick the highest-passing candidate or escalate if none passes. Distinct from multi-path-plan-generator (which generates candidates without mandating verification). Pair with strategic-preparation-phase (which provides the constraint list), planner-executor-verifier, multi-path-plan-generator.

What this pattern forbids. No candidate is committed without passing the Test step; the constraint list is declared up front, not invented during generation.

And the patterns that stand alongside it, or against it —

  • complementsMulti-Path Plan Generator★★— Generate multiple candidate next-steps at each plan node enabling later selection — the planning generator pattern paired with tree-of-thoughts / LATS-style search.
  • complementsStrategic Preparation Phase★— Mandate an explicit problem-space representation step before the agent attempts solutions, mirroring how expert humans build a mental model of constraints and dependencies before solving.
  • complementsPlanner-Executor-Verifier (PEV)★— Triadic specialization where a planner produces the plan, an executor runs it, and a separate verifier checks each step's effects against the original goal.
  • complementsBest-of-N Sampling★— Sample N candidate outputs and select the highest-ranked by a reward model or scorer.
  • alternative-toPremature Closure✕— The LLM commits to a confident answer before processing all constraints, characteristic of constraint-heavy tasks where it fills in plausible answers fast and gets cross-constraint interactions wrong.
  • alternative-toContext Fragmentation✕— Anti-pattern: the LLM cannot hold multiple interconnected constraints in mind simultaneously the way human working memory can; it processes each constraint locally and loses the cross-constraint view.
  • complementsLarge Reasoning Model (LRM) Paradigm★— Route reasoning-heavy tasks to a reasoning-tuned model that trades inference time for deliberation, rather than to a fast LLM that exhibits premature-closure.

Neighbourhood

Click any neighbour to follow the language. Scroll to zoom, drag to pan.

References

Provenance