Overview

Relevant source files

• .github/banner_dark.png
• .github/banner_light.png
• README.md
• examples/voice_agents/push_to_talk.py
• examples/voice_agents/resume_interrupted_agent.py
• livekit-agents/livekit/agents/__init__.py
• livekit-agents/livekit/agents/llm/chat_context.py
• livekit-agents/livekit/agents/version.py
• livekit-agents/livekit/agents/voice/__init__.py
• livekit-agents/livekit/agents/voice/agent.py
• livekit-agents/livekit/agents/voice/agent_activity.py
• livekit-agents/livekit/agents/voice/agent_session.py
• livekit-agents/livekit/agents/voice/audio_recognition.py
• livekit-agents/livekit/agents/voice/events.py
• livekit-agents/livekit/agents/voice/generation.py
• livekit-agents/livekit/agents/voice/run_result.py
• livekit-agents/livekit/agents/voice/speech_handle.py
• livekit-plugins/livekit-plugins-anthropic/livekit/plugins/anthropic/version.py
• livekit-plugins/livekit-plugins-assemblyai/livekit/plugins/assemblyai/version.py
• livekit-plugins/livekit-plugins-azure/livekit/plugins/azure/version.py
• livekit-plugins/livekit-plugins-cartesia/livekit/plugins/cartesia/version.py
• livekit-plugins/livekit-plugins-deepgram/livekit/plugins/deepgram/version.py
• livekit-plugins/livekit-plugins-elevenlabs/livekit/plugins/elevenlabs/version.py
• livekit-plugins/livekit-plugins-fal/livekit/plugins/fal/version.py
• livekit-plugins/livekit-plugins-fal/pyproject.toml
• livekit-plugins/livekit-plugins-google/livekit/plugins/google/version.py
• livekit-plugins/livekit-plugins-nltk/livekit/plugins/nltk/version.py
• livekit-plugins/livekit-plugins-openai/livekit/plugins/openai/version.py
• livekit-plugins/livekit-plugins-silero/livekit/plugins/silero/version.py
• livekit-plugins/livekit-plugins-turn-detector/livekit/plugins/turn_detector/version.py
• tests/test_evals.py
• tests/test_nested_agent_task.py

This document introduces the LiveKit Agents framework, a Python library for building realtime, programmable voice AI agents that run on servers. It explains the framework's purpose, architecture, and key components.

For information about installing and running your first agent, see Quick Start. For details on the monorepo structure and version management, see Project Structure and Versioning. For in-depth coverage of the runtime architecture, see Core Architecture.

---

Purpose and Scope

LiveKit Agents is a framework for building conversational, multi-modal voice agents that can see, hear, and understand. The framework provides:

• Abstraction layers for Speech-to-Text (STT), Large Language Models (LLM), Text-to-Speech (TTS), Voice Activity Detection (VAD), and Realtime APIs. livekit-agents/livekit/agents/__init__.py23 livekit-agents/livekit/agents/voice/__init__.py1-20
• Runtime orchestration through process-based job scheduling and worker management. livekit-agents/livekit/agents/worker.py121-125 livekit-agents/livekit/agents/job.py32-39
• Voice pipeline management including turn detection, interruption handling, and audio/text synchronization. livekit-agents/livekit/agents/voice/turn.py114-119 livekit-agents/livekit/agents/voice/audio_recognition.py126-140
• Plugin ecosystem with 60+ provider integrations including OpenAI, Google, Anthropic, Deepgram, and ElevenLabs. livekit-plugins/livekit-plugins-openai/livekit/plugins/openai/version.py15 livekit-plugins/livekit-plugins-deepgram/livekit/plugins/deepgram/version.py15 livekit-plugins/livekit-plugins-elevenlabs/livekit/plugins/elevenlabs/version.py15
• Development tooling for hot reloading and production deployment via the lk-agents CLI. README.md137

The framework handles low-level streaming logic, turn detection, and tool orchestration so developers can focus on defining agent instructions and implementing function tools.

Sources: README.md23-44 livekit-agents/livekit/agents/worker.py121-125 livekit-agents/livekit/agents/voice/__init__.py70-94

---

System Architecture

LiveKit Agents uses a layered architecture organized by functional concerns: deployment infrastructure, agent orchestration, voice interaction processing, I/O management, AI provider abstraction, and plugin ecosystem.

Overall System Architecture and Component Relationships

The architecture separates deployment infrastructure from agent logic, and isolates provider-specific implementations into plugins. AgentSession serves as the central orchestrator binding all layers together.

Sources: livekit-agents/livekit/agents/worker.py121-125 livekit-agents/livekit/agents/voice/agent_session.py200 livekit-agents/livekit/agents/voice/agent_activity.py139-141 livekit-agents/livekit/agents/voice/room_io.py101

Agent Deployment and Execution Flow

This sequence shows the complete agent lifecycle. The AgentServer registers with LiveKit and receives job assignments, delegating execution to isolated JobProcess instances.

Sources: livekit-agents/livekit/agents/worker.py121-125 livekit-agents/livekit/agents/job.py36-39 README.md108-137

Layer Responsibilities

Layer	Key Components	Responsibilities
Deployment	AgentServer, WorkerOptions, JobProcess	Process management, job distribution, worker registration livekit-agents/livekit/agents/worker.py121-125
Orchestration	AgentSession, AgentTask, ChatContext	Conversation lifecycle, state transitions, speech scheduling livekit-agents/livekit/agents/voice/agent_session.py200
Pipeline	AgentActivity, AudioRecognition, TurnHandlingOptions	Audio processing, turn detection, endpointing, interruption detection livekit-agents/livekit/agents/voice/agent_activity.py139-141 livekit-agents/livekit/agents/voice/audio_recognition.py126-140
I/O	RoomIO, AudioInput, AudioOutput	LiveKit room connections, audio/video stream management livekit-agents/livekit/agents/voice/room_io.py101
Abstraction	llm.LLM, stt.STT, tts.TTS	Standard interfaces for multi-provider support livekit-agents/livekit/agents/__init__.py23

Sources: livekit-agents/livekit/agents/worker.py121-125 livekit-agents/livekit/agents/voice/agent_session.py200 livekit-agents/livekit/agents/voice/turn.py114-119

---

Core Components

Component Hierarchy

Sources: livekit-agents/livekit/agents/worker.py121-125 livekit-agents/livekit/agents/voice/agent_session.py200 README.md137

Component Descriptions

worker.AgentServer livekit-agents/livekit/agents/worker.py121-125

• Entry point class that manages worker lifecycle and job registration.

job.JobContext livekit-agents/livekit/agents/job.py34-39

• Provides access to the rtc.Room connection and job metadata.

voice.AgentSession livekit-agents/livekit/agents/voice/agent_session.py200

• Primary high-level API for managing a voice conversation. Coordinates STT, LLM, and TTS components.

voice.Agent livekit-agents/livekit/agents/voice/agent.py36-57

• Represents the AI agent's identity, including its instructions and tools. livekit-agents/livekit/agents/voice/agent.py75-76

llm.ChatContext livekit-agents/livekit/agents/llm/chat_context.py233-247

• Maintains conversation history using ChatMessage objects. Supports system, user, assistant, and tool roles. livekit-agents/livekit/agents/llm/chat_context.py37-41

voice.room_io.RoomIO livekit-agents/livekit/agents/voice/room_io.py101

• Manages audio and text I/O between the agent and the LiveKit room.

Sources: livekit-agents/livekit/agents/voice/agent_session.py200 livekit-agents/livekit/agents/voice/agent.py36-57 README.md71-77

---

Voice Processing Pipeline

The framework implements a voice processing pipeline that handles concurrent audio input processing and speech generation.

Pipeline Flow and Component Interaction

Sources: livekit-agents/livekit/agents/voice/agent_activity.py139-141 livekit-agents/livekit/agents/voice/audio_recognition.py126-140 livekit-agents/livekit/agents/voice/generation.py59-70

---

Multi-Agent Workflows

The framework supports complex multi-agent handoffs through the AgentHandoff and AgentTask abstractions. Tools can return an AgentHandoff object, which triggers the session to transition to a new agent while optionally maintaining conversation history. livekit-agents/livekit/agents/llm/chat_context.py43 livekit-agents/livekit/agents/voice/agent.py228

Sources: README.md146-175 livekit-agents/livekit/agents/voice/agent_session.py200