elevenlabs-agents

What it does

Production guidance for building voice (and text) agents on the ElevenLabs Agents platform. It guides selecting and tuning the LLM, TTS voice/model, and ASR; configuring the system prompt and first message; attaching knowledge base documents; wiring channels and tools; and evaluating latency, throughput, and governance considerations for real-time, two-way conversations.

How it works

The skill describes a four-component conversation loop: Speech-to-Text (ASR), Turn-taking/VAD, LLM, and Text-to-Speech (TTS). It recommends latency-focused choices (e.g., eleven_flash_v2_5 for TTS, fast LLMs for quick first tokens) and provides a cascading fallback among LLMs (Gemini 2.5 Flash → Gemini 2.0 Flash → Claude 3.7 Sonnet → Claude 3.5 Sonnet v2). It instructs configuring the system prompt for spoken output (brevity, no markdown, persona, edge handling, tool trigger prose) and selecting an LLM from a list of OpenAI, Google, Anthropic, ElevenLabs-hosted weights, or custom endpoints. It details voice/TTS options (eleven_flash_v2_5 as default, with multilingual and higher-quality options), language capabilities, and latency guidance. It explains knowledge base attachment (sources by file/URL, size limits, and how RAG is enabled). It covers tool types (server/webhook, client tools, system tools) and the need for idempotent design and latency masking. It describes personalization via dynamic variables and overrides, and outlines multi-agent workflows, transfers, and channel/SDK support. It emphasizes configuring per-conversation prompts and maintaining testing and simulation.

When to use it

Use when creating a two-way spoken (or typed) conversation driven by the ElevenLabs Agents runtime, including phone bots, voice widgets, in-app assistants, or integrations that require prompt design, voice selection, channel wiring, and evaluation. Do not use for plain one-shot text-to-speech, dubbing, or music generation.

What it can touch

• LLM selection and cascading fallbacks
• Voice and TTS models (e.g., eleven_flash_v2_5, eleven_multilingual_v2)
• ASR, VAD, and turn-taking configuration
• Knowledge base attachments (PDF/TXT/DOCX/HTML/EPUB) and RAG enablement
• Tools: server/webhook tools, client tools, and system tools
• Personalization via dynamic variables and overrides
• Channel SDKs and transport options (WebRTC/WebSocket, embeddable widget, Twilio/SIP)
• Agent workflows, subagents, and human transfers

Caveats

• Models, prices, limits, and endpoints are volatile and should be re-verified against live ElevenLabs documentation before quoting or implementing.
• The guidance emphasizes latency vs. quality trade-offs and cautions that end-to-end latency depends on multiple components, including LLM and telephony.
• Embedding model details and internal retrieval specifics for knowledge bases are not published.
• The content warns that features and availability vary by region (EU residency affects some models).