Learnability-Frontier Task Sampling

Problem

Most of an inherited pool carries no training signal at any given moment. Measurements across two optimisation algorithms and two widely used datasets found that many questions are solved by every attempt, meaning the skill is already learned, or by none, meaning no attempt reaches a reward at all. When every attempt in a group shares the same outcome, a group-relative advantage is zero for every member, so the update contributed by that group is nothing. Uniform sampling therefore spends most of an expensive rollout budget re-confirming what the agent already does and hammering tasks it cannot yet reach, and the fraction of wasted budget grows as training proceeds and more of the pool becomes solved.

Solution

Treat the task pool as a scheduled resource rather than a fixed dataset. Each task carries a running success rate estimated from its recent attempts, and a batch is drawn only from the band where that rate is neither zero nor one — the tasks the agent solves sometimes, whose groups still produce a non-zero advantage. Tasks above the band are retired as learned; tasks below it are parked rather than deleted, and a small share of each round re-probes the parked set so a difficulty step that later becomes reachable is picked up instead of lost. Because the frontier moves as the policy improves, the band is recomputed every round and new tasks are generated from recent unsuccessful attempts, which keeps supply in the band once the inherited pool has been exhausted. A curation step prunes redundant or low-utility items under a cost-aware objective so the pool does not grow without bound. The selector reads only outcomes the same verifier produced for the same served policy, so the band describes the model actually being trained.

When to use

• The agent is trained with a group-relative outcome objective where a uniform-outcome group produces no gradient.
• A single rollout is expensive enough that the choice of task materially changes what the run learns per hour.
• The task pool is large and inherited, so its difficulty distribution was never matched to the current policy.
• Automatic outcome checking is reliable enough that a per-task success rate means competence rather than grader noise.