III · Tool Use & EnvironmentEmerging★

Foveated Perception Escalation

also known as Global-to-Local Visual Allocation, Coarse-to-Fine Zoom, Budgeted Fidelity Restoration, Adaptive Resolution Perception

Treat observation fidelity as a budget the agent allocates: perceive the whole surface coarsely, let the reasoning name the region that decides the answer, then re-observe only that region at full resolution.

This pattern helps complete certain larger patterns —

used-byComputer Use★— Let the model drive a desktop end-to-end via screenshots plus virtual mouse/keyboard tool calls instead of bespoke per-app APIs.
used-byMobile UI Agent★— Drive a smartphone end-to-end through a small, touch-native action vocabulary (tap, long-press, swipe, type, back, home) over screenshots, as a distinct interaction surface from desktop Computer Use and from web Browser Agents.

Context

An agent works from rendered surfaces — a scanned page, a dense screenshot, a history of past screens. The cost of such an observation grows with pixel area, because a vision-language model turns image area into tokens, so a full-acuity rendering of every page or every archived frame either does not fit the active window or costs more than the task is worth. The usual response is a single fixed downscale applied uniformly before the model ever sees the image, and a rule that only the most recent few frames keep their pixels at all.

Problem

A uniformly downscaled observation loses exactly the detail some questions depend on — a figure caption, a small toggle, the state of a control that was clicked twenty steps earlier — and it loses it silently. The model still answers, in the same fluent register it would use at full acuity, and nothing in the output marks resolution as the limiting factor, so the trace shows a confident wrong answer rather than a legibility failure. Raising the global resolution to cover the worst case pays the full price on every observation, and almost none of them needed it.

Forces

Vision token cost grows with pixel area, so full acuity over a whole page, screen or screenshot history is unaffordable at long horizons, while the area that decides the answer is usually a small fraction of the surface.
A coarse observation yields a confident answer rather than an abstention, so an acuity failure leaves no signal at the point where it happens.
Which region needs acuity is question-dependent and cannot be fixed in advance; only reasoning over the coarse view can name it.
Every escalation is another observation round trip, so an agent that zooms at every step spends the latency the coarse pass was meant to save.
Restoring fidelity to an archived observation competes with the newest one for the same budget, and recency is the cheap default that is often, but not always, the right allocation.
A finding read off a crop is expressed in crop coordinates, so the escalation only pays off if the transform back to the original frame is kept and applied.

When to use

The observation is a rendered surface whose token cost scales with area, such as a document page, a dense screenshot, or a history of past screenshots.
The detail that decides the answer occupies a small fraction of the surface, and which fraction depends on the question.
A coarse view is enough to locate the region even when it is not enough to read it.
The surface can be re-rendered on demand at a chosen scale, and the crop transform can be inverted to recover original coordinates.
The task runs long enough that observation cost, not reasoning cost, is what limits the horizon.

When not to use

The whole surface fits the budget at full acuity, so a second look only adds round trips.
The answer depends on the surface as a whole, such as an overall layout or style judgement, and no single region can be nominated.
The source cannot be re-observed because the image arrived once at a fixed resolution and no higher-fidelity original is stored.
The latency budget leaves no room for a second observation round trip before the action must be taken.
Interface state is available as structured accessibility data, which gives exact targets without any pixel budget at all.

Example

An assistant is asked which lab value in a 60-page scanned report is marked abnormal. Rendering every page at full resolution would not fit its window, so it reads all the pages at low resolution and finds the one table that carries flags. It then re-renders just that table at full resolution, reads the flagged row, and reports the value together with the page it came from.

Diagram

flowchart TD A["Observe whole surface at low fidelity"] --> B["Reason over the coarse view"] B --> C{"Deciding detail legible?"} C -- "yes" --> H["Answer or act"] C -- "no" --> D["Nominate region, scale, or archived event"] D --> E{"Fidelity budget left?"} E -- "no" --> G["Abstain or escalate to a human"] E -- "yes" --> F["Re-render only the nominated area at full resolution"] F --> I["Compose crop transform; map findings back to the original frame"] I --> B

Solution

Therefore:

Split perception into a cheap global pass and a small number of expensive local ones. The first pass renders the entire surface at low resolution, enough to see layout, structure and rough positions but not enough to read fine detail; optional coordinate anchors such as detected text boxes and interface elements make that coarse view good enough to point with. The reasoning then either answers, or declares the deciding detail illegible and nominates a target: a region and scale on the current screen, a page in a document, or an archived frame whose pixels should come back. A budget caps how many such escalations a task may spend, on the spatial axis and on the temporal one, and each new view has to be strictly finer than the one it came from so the loop narrows instead of drifting. The nominated area is re-rendered at full resolution, the composed crop transform is kept, and any coordinate the model returns is mapped back to the original frame before an action uses it. When the budget runs out and the detail is still not legible, the loop abstains or escalates rather than answering from the coarse view.

What it gives you

The expensive rendering is paid only for the area the question depends on: coarse-to-fine cropping raised interface grounding accuracy by up to 24.9 percentage points, and a document agent that treats resolution as a reasoning-time resource gained 4.3 to 16.4 accuracy points while cutting inference latency by 41 to 68 percent.
The loop gains an explicit way to say that a detail is not legible at this scale, which turns a silent acuity failure into a further observation; the same document agent reduced hallucination on long documents by more than 40 percent.
Applying the budget along the time axis lets an older observation outbid a recent one: restoring archived screenshots by predicted utility beat spending the whole visual budget on recency, 36.8 percent against 30.2 percent across 117 mobile tasks.
Fidelity becomes an accountable line item, so observation cost per solved task can be measured and capped instead of being fixed at authoring time.

What it costs you

The nomination step can name the wrong region, and a confident answer read off a sharp crop of the wrong area is harder to catch than an obviously blurry one.
Each escalation costs a round trip, and on short horizons the selector does not earn it: under a 15-step limit, utility-scored fidelity restoration and plain recency were statistically indistinguishable.
The gain concentrates where the answer really depends on a distant region or event; on single-application control tasks the same machinery showed no detectable difference, 43.6 percent against 42.4 percent.
Coordinates from a crop must be mapped back through the composed transform, and an off-by-scale mapping produces a click that lands near the target but not on it.
Teaching an agent to zoom well is its own cost: the document agent needed 17.9K supervised zoom trajectories plus 19.2K reinforcement examples before the behaviour was reliable.

What this pattern forbids. The agent must not render the whole surface at full fidelity: a high-resolution observation is permitted only for a region the reasoning nominated from a coarse view, at most a fixed budget of such escalations per task, each view strictly finer than the one before it, and a finding taken from a crop cannot be acted on before its coordinates are mapped back to the original frame.

And the patterns that stand alongside it, or against it —

complementsDual-System GUI Agent★— Split a GUI agent into a decision model that plans and recovers from errors and a grounding model that observes pixels and emits the precise action; route each subproblem to the better-suited model.
complementsPolicy-Localizer-Validator★— Split a GUI agent into three specialist models — a Policy that plans, a Localizer that grounds elements to pixels, and a Validator that judges completion — so each role uses the smallest sufficient model.
complementsAdaptive Compute Allocation★— Allocate inference-time compute (thinking tokens, samples, depth, model size) per query based on input difficulty, rather than using a fixed budget across all queries.
complementsHierarchical Retrieval★★— Route a query through a multi-level cascade — coarse source or index selection, then per-source narrower retrieval, then chunk-level — so each retrieval decision is pushed to the cheapest tier that can answer it.
complementsTool-Result Eviction★— Once a tool's raw output has been consumed, replace it in the live context window with a short marker of what was done, reclaiming tokens without losing that the call happened.
complementsModality-Conflict Arbitration★— Have each observation channel emit its own recorded claim, treat disagreement between channels as a detected event, and resolve it against a pre-declared per-claim-type modality authority rather than an implicit fused guess.