Cumulative Disclosure Accounting

also known as Session Privacy Budget, Compositional Leakage Control, Composition-Aware Query Gate

Keep a session-scoped ledger of what a data agent has already disclosed and evaluate each new query against that accumulated record, so a sequence of individually-permitted results cannot jointly re-identify protected rows.

Context

An agent has read access to a sensitive dataset and answers questions by issuing many queries against it over the course of a task. A policy layer sits between the agent and the database and decides whether each query may run. Every result the agent gets back is an aggregate, a redacted extract, or a filtered slice that on its own satisfies the policy — no identifier is returned, and no group is smaller than the declared minimum. Over a single session the agent may issue dozens of such queries, and their results all reach the same reader.

Problem

The gate decides one call at a time and keeps no record of what it has already released, so it cannot see the join. Four aggregates that are each computed over a large enough group can intersect on a group of one, and the person the policy was written to protect is disclosed by arithmetic the gate never performed. Classical database-privacy work names this a composition attack and shows experimentally that it breaches privacy in practice against a large class of published anonymisation techniques, k-anonymity and several of its variants among them. A layered study of data agents finds the same hole in the policy layer of current systems, which approve each query in isolation and never track cumulative disclosure across the session.

Forces

Independently anonymised releases are each within policy and jointly are not; experiments on composition attacks breach privacy for a large class of published techniques, including k-anonymity and several of its variants.
A per-call gate is cheap and stateless, while cumulative accounting needs durable per-principal state and a model of how results join, which the read path must carry on every query.
A systematic study of data agents identifies eight agent-specific risks across the interpretation, execution and policy layers and, across six evaluated systems — four open-source data agents and two production cloud analytics services — finds policy checked one tool invocation at a time.
Randomised mechanisms such as differential privacy provably resist composition and arbitrary side information, but they add noise to every answer, so accuracy falls as the account is spent.
Among the governance mechanisms surveyed for agents working over data, only information-flow control covers both compositional and cross-session inference leakage, and those two are the least-protected risks in the field.

When to use

An agent issues many reads against the same sensitive dataset within one task or session.
Results are aggregates or redacted extracts that are individually within policy but can be joined by whoever receives them.
A data owner needs to state how much has been disclosed to a principal, not only that each release was permitted.
The dataset covers a population containing groups small enough to be re-identified from a handful of slices.

When not to use

Requests are genuinely independent and their results cannot be joined, such as single-record lookups the caller was already entitled to read in full.
Latency or throughput requirements rule out durable per-session state on the read path.
The data is public or synthetic, so a composed inference discloses nothing the policy protects.
No stable principal identity exists, in which case the account can be reset at will and offers false assurance.

Example

A hospital analyst asks a data agent for the average length of stay by department, then by age band, then by postcode, and finally for the same figure restricted to one rare diagnosis. Each answer covers more than the minimum group size, so the per-call policy allows all four. Read side by side, the four results leave exactly one patient who can be in the last cell, and the agent has just published that person's stay. A session ledger would have refused the fourth query because it narrowed a group the first three had already bounded.

Diagram

sequenceDiagram participant A as Data agent participant G as Disclosure gate participant L as Session ledger participant D as Database A->>G: query 1, stay by department G->>L: charge disclosure L-->>G: within account G->>D: execute D-->>A: aggregate result 1 A->>G: query 2, same metric by age band G->>L: charge disclosure L-->>G: within account G->>D: execute D-->>A: aggregate result 2 A->>G: query 3, same metric for a rare diagnosis G->>L: join proposal with results 1 and 2 L-->>G: group narrows to one row, account overdrawn G-->>A: refusal, escalate to the data owner

Solution

Therefore:

Make the policy decision stateful. Open an account when a principal starts a session, and record in a ledger every result the gate lets out: which table and attributes it touched, which population it was computed over, and what privacy cost it carried. When the next query arrives, the gate does not ask whether this query alone is permitted; it asks what the proposed result would reveal once joined with everything the ledger already holds. Two mechanisms answer that question. A composition check tests whether the new result intersected with prior ones narrows any protected group below the declared minimum, and refuses or coarsens the query when it does. A privacy budget prices each query in a randomised mechanism whose composition bound is provable, subtracts the price from the session's remaining allowance, and stops when the allowance runs out. Every disclosure channel is metered, not only direct table reads — tool results, memory writes, and messages passed to another agent all spend from the same account. Exhaustion is surfaced as an exhausted account escalated to the data owner, never as a quietly truncated answer, and the ledger is keyed to the principal so that opening a fresh session does not refill it.

What it gives you

A run of individually safe releases can no longer compose into a re-identification, because the decision is made on the join rather than on the single call.
The data owner gets one number for how much has been disclosed to a principal, in place of an unbounded stream of separately approved reads.
A refusal names an exhausted account rather than a rule the caller can route around by rephrasing the same question.
Cross-session leakage becomes visible, because the account is keyed to the principal and survives the end of a session.

What it costs you

The gate stops being stateless, so the ledger becomes a durability and correctness dependency, and a lost or unavailable ledger silently degrades the system back to per-call decisions.
An account that binds the whole session eventually stops legitimate work, and the tighter the allowance the sooner an honest analyst hits it.
Thresholds on cell counts do not compose, so a ledger built only on minimum group size still admits breaches that only a randomised mechanism provably resists.
Every channel out of the agent has to be metered, and one unmetered channel — an intermediate tool result, a memory write, a message to a peer agent — voids the account without any signal that it did.
Pricing a query needs a model of what it reveals, and that model is dataset-specific work that has to be maintained as schemas change.

What this pattern forbids. A query must not be approved on its own merits alone: the gate must first charge the proposed disclosure against the principal's session ledger and refuse or coarsen any query whose result, joined with what the session has already returned, would narrow a protected group below the declared minimum or overdraw the remaining budget. A new session must not reset an account bound to the same principal.

The smaller patterns that complete this one —

usesProvenance Ledger★★— Log every agent decision and state change with enough metadata to explain or reverse it later.

And the patterns that stand alongside it, or against it —

complementsPII Redaction★★— Detect and remove personally identifiable information from inputs to and outputs from the model.
alternative-toRate Limiting★★— Cap the number of requests, tokens, or tool calls per user (or session) within a time window.
complementsSession-Scoped Payment Authorization·— Bound an agent's autonomous spending by having it open a payment session with a pre-approved cap, stream many micropayments inside that session, and settle once on close, instead of seeking approval for every transaction.
complementsPolicy-as-Code Gate★— Evaluate every proposed agent action against externally-managed machine-readable policies before dispatch, so compliance authorship lives outside the prompt and outside the agent code.
complementsSemantic-Layer Query Guardrail★— Route natural-language data questions through a curated semantic layer so the model selects and parameterises vetted metrics and dimensions instead of free-authoring raw SQL against production data.
complementsSession Isolation★★— Keep one user's session state and memory unreachable from another user's agent.
complementsLethal Trifecta Threat Model★— Block prompt-injection-driven exfiltration by ensuring no single agent execution path holds all three of: access to private data, exposure to untrusted content, and an outbound communication channel.
complementsMemory Extraction Attack✕— Anti-pattern: let any session prompt the agent to read out, summarise, or paraphrase long-term memory entries belonging to other users, prior sessions, or system state, with no read-time isolation by principal.