voice-agents

What it does

Describes two architectures for voice agents: speech-to-speech for lowest latency and pipeline for more control. It defines latency goals (end-to-end under 800ms, TTFT under 300ms, etc.), outlines system patterns for speech processing, and lists models and components (STT, TTS, LLM) across various toolings. It also presents patterns for voice activity detection, latency optimization, and conversation design to manage turn-taking, barge-in, and error handling.

How it works

The skill instructs the agent to prefer a low-latency, speech-to-speech setup when maximum naturalness is desired, with sub-500ms end-to-end responsiveness and streaming to reduce perceived delays. It provides concrete example configurations and code sketches for the OpenAI Realtime API session and for a separate Pipeline Architecture, including how to wire STT (e.g., Deepgram Nova-3, Whisper), LLM (e.g., gpt-4o-mini), and TTS (e.g., ElevenLabs, Deepgram Aura-2). It details VAD implementations (Silero VAD, Semantic VAD) and barge-in handling, plus latency optimization techniques such as streaming results, pre-computation, edge deployment, and model selection. It also includes design guidelines for voice-first prompts, response length, and error recovery strategies.

When to use it

• When aiming for real-time voice conversations with strict latency targets (end-to-end < 800ms, TTFT < 300ms). - When wanting to minimize jitter and optimize barge-in handling and VAD quality. - When evaluating whether to use Speech-to-Speech vs Pipeline architectures based on debugging needs, control, and latency constraints.

What it can touch

• It references tools and frameworks: OpenAI Realtime API, Pipecat, Deepgram Nova-3, ElevenLabs, Silero VAD, gpt-4o-mini, etc. The explicit sections show where to configure sessions, streaming behavior, and VAD/turn-detection settings. The content implies integration with audio streams, VAD processing, STT, LLM, and TTS pipelines.

Caveats

• Emphasizes latency targets and trade-offs (lower latency vs more control). - Notes that end-to-end latency is influenced by multiple components (VAD, STT, LLM, TTS) and provides ranges (e.g., 425-900ms total). - Contains operational guidance like keeping models warm, streaming outputs, and edge deployment, but does not guarantee specific outcomes or success metrics beyond stated targets.