X · Governance & ObservabilityExperimental·

Multi-Principal Welfare Aggregation

also known as Multi-Principal Assistance Game, Social-Choice Aggregation for Agents

When an agent serves multiple humans with conflicting preferences, declare the aggregation rule explicitly rather than letting it be implicit in the prompt or fine-tune.

Context

An agent serves a team, a household, a customer cohort, or an entire user base. The principals have conflicting preferences: different staff want different summary styles, different customers want different escalation defaults, different users in a shared workspace want different behaviours. Some preferences are zero-sum.

Problem

Without an explicit aggregation rule the agent silently picks one principal — usually the loudest, the most recently heard, or the one whose preferences were fine-tuned in earliest. Gibbard's theorem says any aggregation rule that aggregates more than two principals' preferences is manipulable: principals can strategically misreport. Pretending there is no aggregation rule does not avoid this; it picks the implicit rule and hides it from review.

Forces

  • Multiple principals with conflicting preferences is the common case at scale.
  • Every aggregation rule has trade-offs; none is uniformly best.
  • Hidden aggregation is gameable and unaccountable.
  • Explicit aggregation invites disputes that hidden aggregation avoided.

Example

A household assistant must schedule shared resources (the family calendar, the thermostat). Two adults have conflicting work-from-home preferences. The product declares a weighted-welfare rule with declared weights and exposes them in settings. When the rule produces an outcome one adult dislikes, the dispute is about the weights and the rule, not about the agent's hidden disposition.

Diagram

Solution

Therefore:

When the agent's action space affects multiple principals, route the decision through an explicit aggregation function. Options: sum-of-utilities (utilitarian); weighted welfare (declared per-principal weights); collegial mechanism (each principal must be obtaining 'enough' reward through their own actions for their preferences to count); role-priority (some principals have veto). Surface the active rule in traces and documentation. Make it a configuration change, not a prompt change.

What this pattern forbids. An agent serving multiple principals must not aggregate their preferences implicitly; the aggregation rule is declared as configuration and surfaced in traces.

The smaller patterns that complete this one —

  • usesCooperative Preference Inference·— Agent and human jointly optimise the human's reward without the agent being told what it is — the interaction is a two-player game in which alignment is learned while acting.

And the patterns that stand alongside it, or against it —

  • complementsPreference-Uncertain Agent·— Agent treats its own reward/objective as a hidden variable to be inferred from human behaviour, not a fixed target.
  • composes-withPolicy-as-Code Gate★— Evaluate every proposed agent action against externally-managed machine-readable policies before dispatch, so compliance authorship lives outside the prompt and outside the agent code.
  • complementsDecision Log★★— Persist the agent's reasoning trace alongside its actions so post-hoc review can explain why.
  • complementsTrust and Reputation Routing★— Maintain a per-agent reputation score updated from outcome quality and peer feedback, and route new tasks preferentially to high-reputation agents.

Neighbourhood

Click any neighbour to follow the language. Scroll to zoom, drag to pan.

Used in recipes

References

Provenance