openai-realtime-voice

What it does

Guides a live spoken-interaction agent (browser voice assistant, phone-like agent, or live voice UI) to listen, reason, speak, interrupt, and potentially call tools within a Realtime session. It is not intended for batch transcription, offline TTS, or file-based audio tasks. It emphasizes low-latency, turn-taking, tool use, logging, consent, privacy, safety, latency, and cost controls.

How it works

• The skill targets production OpenAI Realtime voice sessions (live audio with low latency) and distinguishes them from batch or transcription workflows.
• It instructs choosing between architecture options: use Realtime voice-agent session for latency-sensitive, interruptible conversations with tool calls; or a chained STT -> text -> TTS pipeline when deterministic transcripts are priority.
• It prescribes using WebRTC for browser/mobile media transport and WebSocket for server-to-server integration, with session management via a Session object, a Conversation, and Responses. It notes that the voice cannot be changed after the first audio response in a session and that the maximum session duration is 60 minutes.
• It details input-audio handling (base64 via input-audio-buffer for WebSocket; WebRTC handling media), VAD configuration (server_vad or semantic_vad) and tuning (silence_duration_ms, threshold, prefix_padding_ms).
• It allows attaching tools at session or per-response level, including MCP tools, and outlines cost accounting (audio tokens) and data handling expectations (transcripts as observability artifacts, not exact reasoning inputs).
• It provides extensive prompts design guidance for spoken agents, including role, voice persona, turn length, safety, privacy, and escalation rules. It covers turn-taking state machine, interruption rules, and explicit UI controls for stopping/muting.
• It defines when to use tools, how to validate and return results, and safety/approval policies for high-impact actions.

When to use it

Use this skill when the deliverable is a live spoken interaction with real-time turn-taking, tool usage, and user-centric audio UX (e.g., browser voice assistant, low-latency tutoring conversation, live kiosk). Do not use for batch speech-to-text, offline TTS, or audio file editing tasks.

What it can touch

• Tools: function tools, MCP tools, and built-in connectors may be attached at session level or for a single response.
• Media and transport: WebRTC for browser/mobile, WebSocket for server-to-server transport.
• Session/state: model, voice, and configuration managed within a Realtime Session object; conversation items and responses tracked.

Caveats

• OpenAI model names, session limits, and data handling policies are subject to change; producers should verify current references before release.
• Data may be logged (sessions, tool calls, IDs) per the guidance; transcripts are treated as observability artifacts, not exact reasoning inputs.
• VAD behavior is configurable and may require careful tuning for different environments and languages; invalid or disabled VAD places more responsibility on application-side turn boundaries.
• Some capabilities depend on external tool servers or remote connectors and require proper authorization, logging, and privacy measures.