Learnability-Frontier Task Sampling

also known as Sampling for Learnability, Success-Variance Curriculum, Self-Evolving Curriculum

Spend the training rollout budget on tasks the agent currently solves sometimes, dropping tasks that every attempt solves and tasks that no attempt solves because both yield a zero-variance group and no gradient.

Context

An agent is being improved by reinforcement learning against a pool of tasks with automatic outcome checking. The trainer samples a task, runs several attempts, and turns the spread of outcomes within that group into an update. In agent settings a single attempt is not a short completion but a multi-minute episode inside a sandbox, with tool calls, a browser or a shell, so the number of attempts a run can afford is small and fixed in advance. The task pool is usually inherited whole from a benchmark or a scrape and sampled uniformly.

Problem

Most of an inherited pool carries no training signal at any given moment. Measurements across two optimisation algorithms and two widely used datasets found that many questions are solved by every attempt, meaning the skill is already learned, or by none, meaning no attempt reaches a reward at all. When every attempt in a group shares the same outcome, a group-relative advantage is zero for every member, so the update contributed by that group is nothing. Uniform sampling therefore spends most of an expensive rollout budget re-confirming what the agent already does and hammering tasks it cannot yet reach, and the fraction of wasted budget grows as training proceeds and more of the pool becomes solved.

Forces

A fixed pool is cheapest to sample uniformly, but under group-relative advantage a task whose attempts all agree contributes exactly zero to the update, so uniform sampling converts most of the budget into no gradient.
Success variance is the signal that identifies an informative task, yet estimating it costs the same rollouts the selection is meant to conserve.
The informative band moves during training: a task at the frontier this round is fully solved a few rounds later, so any curriculum fixed in advance decays into the uniform case it replaced.
In agent training a rollout is a multi-minute sandboxed episode rather than a token completion, so the price of an uninformative sample is orders of magnitude higher than in ordinary preference training.
Dropping tasks that nobody solves protects the budget but also removes exactly the tasks whose capability is missing, so an aggressive filter can park a difficulty step the agent then never practises.

When to use

The agent is trained with a group-relative outcome objective where a uniform-outcome group produces no gradient.
A single rollout is expensive enough that the choice of task materially changes what the run learns per hour.
The task pool is large and inherited, so its difficulty distribution was never matched to the current policy.
Automatic outcome checking is reliable enough that a per-task success rate means competence rather than grader noise.

When not to use

The verifier is noisy or partially subjective, in which case measured variance reflects grading instability and the band fills with unscoreable tasks.
The pool is small enough to sample exhaustively each round, where the bookkeeping costs more than it saves.
The reward is dense and shaped, so even a task no attempt completes still returns a usable gradient and does not need to be parked.
Training is supervised fine-tuning on fixed demonstrations rather than sampled attempts, since there is no per-task success rate to estimate.

Example

A team trains a browsing agent on two thousand scraped web tasks, and each attempt takes several minutes inside a sandboxed browser. After two rounds they check the per-task numbers and find that a third of the pool is solved by every attempt and about half by none, so roughly four in five episodes changed nothing. They restrict the next batches to the tasks that succeed some of the time and start generating replacement tasks from the failures, and the same nightly budget begins to move the score again.

Diagram

flowchart TD A["Task pool"] --> B["Per-task success rate over recent attempts"] B --> C{"Rate"} C -- "all attempts pass" --> D["Retire: already learned"] C -- "sometimes passes" --> E["Frontier band -> sample batch"] C -- "no attempt passes" --> F["Park; re-probe on a schedule"] F -.-> B E --> G["Run rollouts, verify outcomes"] G --> H["Policy update from mixed-outcome groups"] G --> B G --> I["Generate new tasks from failures"] I --> A

Solution

Therefore:

Treat the task pool as a scheduled resource rather than a fixed dataset. Each task carries a running success rate estimated from its recent attempts, and a batch is drawn only from the band where that rate is neither zero nor one — the tasks the agent solves sometimes, whose groups still produce a non-zero advantage. Tasks above the band are retired as learned; tasks below it are parked rather than deleted, and a small share of each round re-probes the parked set so a difficulty step that later becomes reachable is picked up instead of lost. Because the frontier moves as the policy improves, the band is recomputed every round and new tasks are generated from recent unsuccessful attempts, which keeps supply in the band once the inherited pool has been exhausted. A curation step prunes redundant or low-utility items under a cost-aware objective so the pool does not grow without bound. The selector reads only outcomes the same verifier produced for the same served policy, so the band describes the model actually being trained.

What it gives you

Rollout budget is spent on groups with mixed outcomes, which are the only groups a group-relative estimator turns into a non-zero update.
Prioritising questions with high success variance was reported to improve on uniform sampling under both optimisation algorithms and both datasets it was measured on.
The per-task success rate is a progress read-out for free: the band's composition over time shows which skills have been learned and which are still out of reach.
Generating new tasks from unsuccessful attempts keeps the informative band supplied after the inherited pool has been mined out, which is what makes the curriculum survive policy drift.

What it costs you

Estimating the success rate consumes rollouts that produce no update unless those same attempts are reused as training data.
A band that is too narrow starves the batch and a band that is too wide degenerates into uniform sampling, so the two bounds become hyper-parameters that need tuning per task family.
Parking every task that no attempt solves can strand a capability step permanently if the re-probe schedule is dropped for cost reasons.
Variance caused by a noisy verifier is indistinguishable from variance caused by genuine partial competence, so an unreliable grader pulls the curriculum toward tasks it cannot score rather than tasks worth learning.
The scheduler adds per-task state that has to be persisted, versioned and reconciled across distributed rollout workers.

What this pattern forbids. A training batch must not be drawn uniformly from the task pool: a task whose recent attempts all succeeded or all failed cannot enter the batch until its measured success rate re-enters the frontier band, and no task may be admitted without a current per-task success estimate produced by the same verifier and the same served policy configuration as the rollouts.

And the patterns that stand alongside it, or against it —

complementsReST-EM★— Iterate generate → reward-filter → fine-tune to bootstrap reasoning capabilities without human-labelled data.
alternative-toChecklist Partial-Credit Scoring★— Score each long-horizon task against task-specific graded checkpoints instead of a single pass/fail bit, so progress and regression that leave the binary outcome unchanged still show up in the report.
complementsMirrored Tool Environment★— Replace the live external tool during agent training and benchmarking with a fine-tuned model that emits responses in that tool's shape, so runs stay cheap, reproducible and controllable in difficulty.
complementsHarness-Native Rollout Capture★— Capture reinforcement-learning rollouts by intercepting the model-call boundary inside the harness the agent already runs in, recording exact prompt and response token ids rather than reconstructing them from the transcript.
complementsServed-Policy Anchoring★— Treat the policy the inference engine actually serves, not the one the training engine computes, as the object being optimised and gated, and measure their probability agreement continuously.
complementsSimulated-User Rollout Evaluation★— Evaluate a conversational agent by pairing it with a second model that plays the user from a persona and a hidden goal, then score the final environment state rather than the transcript.
complementsBayesian Bandit Experimentation★— Replace fixed-split A/B tests between agent variants with a bandit that dynamically reallocates traffic toward better-performing variants based on observed reward, bounding regret from bad variants.

Neighbourhood

Click any neighbour to follow the language. Scroll to zoom, drag to pan.

References

Provenance

Source: patterns/learnability-frontier-task-sampling.md on GitHub · commit c389d0e · view history
Added to catalog: 2026-09-10
Last updated: 2026-09-10
Contribute: open an issue or PR at github.com/agentpatternscatalog/patterns.