LiveKit Agents
Type: full-code · Vendor: LiveKit · Language: Python, Node.js · License: Apache-2.0 · Status: active · Status in practice: mature · First released: 2023-10-19
Open-source realtime agent framework that lets a Python or Node.js process join a LiveKit room as a full participant, with an STT-LLM-TTS pipeline, turn detection, tool calling and worker-based job dispatch.
Description. LiveKit Agents is a framework for building realtime voice, video and multimodal agents that operate as participants in LiveKit rooms. The agent code acts as a stateful bridge between AI models and users: it streams audio through an STT-LLM-TTS pipeline, runs a custom turn-detection model for lifelike conversation flow, handles interruptions, and exposes a function_tool decorator so any LLM can call tools (including forwarding calls to the frontend). Agents run as workers; the agent server boots a 'job' subprocess that joins each room. Plugins cover STT, LLM, TTS and Realtime APIs.
Agent loop shape. Worker-dispatched, room-scoped event loop. A LiveKit worker registers with the agent server; when a room is created the server dispatches a job, the job subprocess joins the room, instantiates an AgentSession with STT, LLM, TTS and turn-detection plugins, and runs the streaming pipeline. Tools defined with @function_tool are exposed to the LLM, including forwardable frontend tool calls.
Primary use cases
- realtime voice agents over WebRTC and telephony
- multi-modal assistants that hear, see and speak
- multi-agent handoff over LiveKit rooms
- outbound calling, transcription and realtime translation
Key concepts
- Agent (docs) — A realtime participant that runs on the server and bridges users with AI models.
- STT-LLM-TTS pipeline (docs) — Streaming audio pipeline with reliable turn detection and interruption handling.
- Turn detection → stop-cancel (docs) — Custom transformer model that detects end-of-turn to reduce barge-ins.
- @function_tool → tool-use (docs) — Decorator that exposes Python functions as LLM-callable tools, forwardable to frontend.
- Worker / job (docs) — The agent server boots a job subprocess per room with load balancing.
- Plugins (docs) — Mix-and-match STT, LLM, TTS and Realtime API integrations.
Patterns this full-code implements —
- ★★Stop / Cancel
Pipeline ships reliable turn detection and interruption handling; custom transformer model detects end-of-turn.
- ★★Tool Use
@function_tool decorator exposes tools to any LLM; tool calls can be forwarded to the frontend.
- ★★Event-Driven Agent
Agent is a realtime participant whose code reacts to room events and audio frames as they arrive.
- ★★Conversation Handoff to Human
A handoff transfers session control from one agent to another; returning a different agent from a tool call triggers automatic handoff and adds an AgentHandoff item to the chat context.
- ★★Session Isolation
Each room gets its own job subprocess; the agent server boots a job subprocess per room with load balancing.
- ★★Model Context Protocol
First-class MCP support via MCPToolset passed to an agent's tools parameter; MCPServerHTTP for remote and MCPServerStdio for local subprocess servers.
- ★Multilingual Voice Agent Stack
Plugins expose multilingual modes (e.g. Deepgram nova-3 language='multi'); realtime translation is a documented use case.
- ★★Mandatory Red-Flag Escalation
LiveKit's handoff pattern enforces escalation to a human automatically when compliance mandates or rule conditions are met, passing the receiving human full context rather than leaving the transfer t…
Neighbourhood
Click any neighbour to follow the lineage. Scroll to zoom, drag to pan.
Alternatives & relatives
- full-code · framework
Pipecat
Peer open Python framework for voice agents.
- low-code · visual builder
Vapi
Hosted alternative.
- full-code · framework
ElevenLabs Conversational AI
Hosted alternative.
- low-code · visual builder
Retell AI
Hosted phone-agent alternative.
- full-code · framework
Hume EVI
Open framework alternative.
Listed as alternative by (5)
References
Provenance
- Last analyzed:
- Last updated:
- Verification status: verified