III · Tool Use & EnvironmentMature★★

Tool Result Caching

also known as Memoised Tools, Idempotent Cache

Cache the result of expensive deterministic tool calls keyed by their arguments so repeat calls within a session return immediately.

This pattern helps complete certain larger patterns —

  • specialisesTool Use★★— Let the LLM produce typed calls against an external toolkit instead of producing free-form text the surrounding system has to parse.

Context

A team runs an agent that calls deterministic lookup or computation tools many times within a single task — fetching the same company profile from four sub-tasks, recomputing the same exchange rate, reading the same immutable document for several reasoning steps. The tools are paid (per-call cost), rate-limited, or simply slow, and the agent has no memory of having called them before.

Problem

Repeat calls on identical arguments pay full latency and full per-call cost every time, even though the result has not changed and the tool author would gladly serve it from a cache. The agent's loop is structured one call at a time and has no awareness of caller history, so the same lookup gets re-fetched whenever a different reasoning step happens to need it. Caches written naively can leak results across users when caller identity is not part of the key.

Forces

  • Cache invalidation: when does the underlying data change?
  • Per-user vs global caches differ on isolation guarantees.
  • Cache hits hide tool latency the agent might benefit from learning about.

Example

An agent that researches companies calls the same `get_company_profile(domain)` tool four times per session because different sub-tasks need it. Latency and per-call cost stack up. The team wraps deterministic tools in a cache keyed on `(tool_name, normalised_args)` with TTLs by tool type; per-user scoping keeps tenant-sensitive results from crossing accounts. Repeat calls return immediately, the underlying tool quota lasts longer, and session latency drops.

Diagram

Solution

Therefore:

Wrap deterministic tools in a cache layered on `(tool_name, normalised_args)`. Set TTLs by tool type. On cache hit, return immediately without invoking the underlying tool. Per-user scoping for tools that read user data; global for read-only public data. Cache keys must include the auth subject (caller identity), not just args; args-only keys leak data when callers change.

What this pattern forbids. Only tools declared deterministic may be cached; nondeterministic tools bypass the cache.

And the patterns that stand alongside it, or against it —

  • complementsSession Isolation★★— Keep one user's session state and memory unreachable from another user's agent.
  • complementsRealtime API When Batchable✕— Anti-pattern: use the realtime/synchronous model API for workloads whose latency budget would permit batching, paying 2–10× the unit cost for no user-visible benefit.

Neighbourhood

Click any neighbour to follow the language. Scroll to zoom, drag to pan.

Used in frameworks

References

Provenance