I · ReasoningEmerging★

Adaptive Compute Allocation

also known as Input-Adaptive Thinking Budget, Per-Query Compute Routing, Adaptive Thinking

Allocate inference-time compute (thinking tokens, samples, depth, model size) per query based on input difficulty, rather than using a fixed budget across all queries.

This pattern helps complete certain larger patterns —

  • specialisesTest-Time Compute Scaling★★— Allocate more inference-time compute (samples, search, deeper thinking) instead of scaling parameters to improve quality.

Context

A reasoning agent or inference router serves queries of widely varying difficulty: simple lookups, moderate multi-step reasoning, hard novel problems. Compute per query is the dominant cost. The trivial policy — fixed budget across all queries — either wastes compute on simple ones or under-serves hard ones.

Problem

Static compute budgets force a single trade-off across all queries. With LLM inference cost dominating production economics, the slack on simple queries is large; the deficit on hard queries is real. Recent work (the 2025 arXiv survey 'Reasoning on a Budget', the 2026 ACM Web Conference paper on adaptive routing) shows that input-conditional allocation can reduce cost without sacrificing quality — but only if there is a reliable signal for per-query difficulty available before commitment.

Forces

  • Compute is expensive; over-allocation wastes; under-allocation produces wrong answers.
  • Per-query difficulty is not always knowable upfront; some signals (self-consistency, model-uncertainty) require partial generation to read.
  • Routing-quality and routing-overhead trade off — a complex router can eat the savings.

Example

A customer-support assistant serves 10M queries/month. Profiling shows ~70% are FAQ-style (one-shot), ~25% are multi-step (need plan+execute), ~5% are genuinely novel (need extended thinking). Current setup uses a fixed extended-thinking budget on every query. The team adds a difficulty estimator: a small classifier scores prompt complexity, routes the 70% to the small fast path with no thinking tokens, the 25% to a moderate budget, the 5% to the full extended-thinking budget. Net inference cost drops 60% with no quality regression on production traffic.

Diagram

Solution

Therefore:

Adopt a per-query budget pipeline: cheap difficulty estimator picks initial budget; partial-output signals (low self-consistency, low model confidence, branching mid-reasoning) trigger budget ramp; hard ceiling on budget per query prevents runaway. Variants include model routing (small model first, escalate on uncertainty), thinking-token budget control, and sample-count adaptation. Distinct from test-time-compute-scaling by being explicitly input-conditional.

What this pattern forbids. Imposes a per-query difficulty estimation step before commitment to a compute level; constrains compute budgets to be elastic per query rather than flat across the deployment.

The smaller patterns that complete this one —

  • generalisesAdvisor Consult★— Let a lower-cost executor model run the whole task and, at decision points it chooses, consult a stronger advisor model that reads the full transcript and returns guidance only, never actions.

And the patterns that stand alongside it, or against it —

  • complementsSleep-Time Compute·— During idle or downtime, run the model offline against the user's standing context to pre-compute dense summaries and likely future answers, so test-time latency and cost drop when the user actually asks.
  • complementsMode-Adaptive Cadence★— Vary the agent's loop interval based on current salience so the agent thinks faster when something is happening and slower when nothing is, instead of running on a fixed cron.
  • complementsMulti-Model Routing★★— Send each request to the cheapest model that can handle it well.
  • complementsProcess Reward Model★— Train a verifier that scores each reasoning step rather than only the final answer.
  • complementsComplexity-Based Routing★— Estimate a request's difficulty up front and bind it to the cheapest model tier that can answer well, using an explicit complexity classifier as the routing key.
  • alternative-toDeliberation Over Observation✕— Anti-pattern: give a reasoning-trained model a large thinking budget in an interactive environment and it hypothesises about the environment's state instead of issuing the cheap observation that would settle it.
  • complementsFoveated Perception Escalation★— Treat observation fidelity as a budget the agent allocates: perceive the whole surface coarsely, let the reasoning name the region that decides the answer, then re-observe only that region at full resolution.

Neighbourhood

Click any neighbour to follow the language. Scroll to zoom, drag to pan.

Used in frameworks

References

Provenance