Routing & Composition

Send each request to the cheapest model that can handle it well.

Try a primary handler; on failure or low confidence, fall through to a sequence of fallback handlers.

Stop calling a failing dependency for a cooldown period after error rates exceed a threshold.

Compose stream-shaped processing as a chain of small filters connected by pipes.

Treat the agent's workflow (a graph of LLM-invoking nodes) as an artefact to search; use Monte Carlo Tree Search guided by an eval benchmark to discover the best workflow, then deploy it.

When a dependency fails, downgrade the user-facing experience to a working subset rather than failing entirely.

When one provider's API errors mid-stream, transparently switch to another provider while preserving state.

Estimate a request's difficulty up front and bind it to the cheapest model tier that can answer well, using an explicit complexity classifier as the routing key.

Route each request to one or more domain-expert agents, where each expert holds deep capability in a narrow area.

Build a multi-model cascade where lower tiers are open-weight, self-hostable models that run inside the operator's boundary, and only escalations cross to a hosted frontier model — giving cost arbitr…

Allow the model to emit several independent tool calls in one assistant turn; the host executes them in parallel.

Run independent LLM calls concurrently and combine results.

Decompose a task into a fixed sequence of LLM calls where each step's output becomes the next step's input.

Classify an incoming request and dispatch it to the specialist (lane / agent / model) best suited to handle it.

Treat agent identity as a structured profile object — persona, primary motivator, allowed actions, knowledge bindings — rather than a free-form role sentence in the system prompt.

Select the model and provider for a request through a single namespaced string (`provider/model`) backed by env-var credentials, so the caller specifies what to run with one parameter rather than a t…

Maintain a per-agent reputation score updated from outcome quality and peer feedback, and route new tasks preferentially to high-reputation agents.

Route each request through an LLM dispatcher to specialized symbolic or neural expert modules (calculator, knowledge base, code executor) rather than asking one LLM to do everything; integrate the mo…

BPMN processes and DMN decision tables form the deterministic spine; LLM-driven agents are invoked only at explicit 'unstructured problem' nodes inside the process.

Inject task-specific scaffolding (examples, hints, schemas) into the prompt only when the task type warrants it.

Per query, route between a symbolic path (rule engine, knowledge graph) and a neural path (LLM), using the LLM for interpretation and the symbolic layer for exact constraints.