Portkey (Conditional Routing)
Type: low-code · Vendor: Portkey AI · Language: TypeScript · License: MIT · Status: active · Status in practice: mature · First released: 2023
Portkey is an AI gateway that routes each LLM request to a provider target by deterministic rules over request metadata and parameters, and falls back to another provider when one fails.
Description. Portkey's AI Gateway sits in front of many LLM providers behind one API. Its conditional routing evaluates custom conditions over request metadata, request parameters, and the URL path to pick a provider target, so a rule path runs ahead of the models. When the primary provider fails, Portkey automatically falls back to the next configured target. It also adds retries, load balancing, and guardrails.
Agent loop shape. Portkey is a gateway, not an agent loop. Each incoming request is matched against conditional-routing rules over its metadata and parameters to select a provider target; the gateway forwards the call, applies retries, and on failure falls back through an ordered list of alternative targets before returning the response to the caller.
Primary use cases
- rule-based routing of requests to provider targets
- automatic fallback across providers on failure
- load balancing across LLM deployments
- gateway-level guardrails and observability
Key concepts
- Configs (docs) — A declarative JSON object attached to a request that composes the gateway's routing behaviour - conditional routing, fallbacks, load balancing, retries, caching, and guardrails - without changing application code.
- Targets → fallback-chain (docs) — The ordered list of provider/model destinations a Config can route to; fallbacks try them in priority order and load balancing splits traffic across them by weight.
- Load balancing weights (docs) — Per-target weights that Portkey normalises to 100% and uses to distribute requests proportionally across multiple LLM deployments, e.g. weights 5,3,1 become 55%,33%,11%.
- Guardrails → input-output-guardrails (docs) — Pluggable checks that run synchronously or asynchronously around the model call to validate requests and responses, with configurable actions on a failed check.
Patterns this low-code implements —
- ★★Fallback Chain
Portkey lets you define a prioritised list of provider/model targets; when the primary fails (by default on any non-2xx, configurable to specific codes) it automatically falls back to the next in lin…
- ★Provider-String Routing
Portkey fronts 200+ models behind one universal API where the model and provider are selected by an @provider/model string; switching providers is a one-parameter change while the request format stay…
- ★Hybrid Symbolic-Neural Routing
Portkey's AI Gateway routes each request to a provider target via symbolic rules over metadata and request params (user plan, model flag, region, temperature) — a deterministic rule path sitting in f…
- ★★Input/Output Guardrails
Portkey can run guardrail checks synchronously on both sides of the model call: before the request reaches the LLM and before the response reaches the user, so unsafe inputs and outputs are validated…
- ★Semantic Response Cache
Beyond exact-match (simple) caching, Portkey offers a semantic cache that returns a stored response when a new query is close to a previous one by cosine similarity above a threshold, so paraphrased…
- ★★Circuit Breaker
When a provider target keeps failing, Portkey opens a circuit and stops routing to it for a cooldown interval, removing the unhealthy target from rotation until it recovers, so the gateway stops hamm…
Neighbourhood
Click any neighbour to follow the lineage. Scroll to zoom, drag to pan.
Alternatives & relatives
- app · closed product
OpenRouter
OpenRouter is also a unified gateway in front of many LLM providers with fallback and routing, but is positioned more as a hosted marketplace than a self-hostable open-source gateway.
- app · closed product
Not Diamond
Not Diamond focuses on quality-based model selection (picking the best model per prompt) rather than Portkey's rule-based conditional routing and reliability layer.
- full-code · framework
RouteLLM
RouteLLM routes between a strong and weak model to cut cost; it overlaps Portkey's routing goal but is a router library rather than a full gateway with caching, guardrails, and fallbacks.
References
“Use any of the supported models with a universal API (REST and SDKs)”
“Prevent downtimes through automatic retries and fallbacks”
“Distribute traffic across multiple LLMs to prevent any single provider from becoming a bottleneck.”
Provenance
- Last analyzed:
- Last updated:
- Verification status: partial