gemini-live-audio

What it does

Guides live, bidirectional voice sessions using Gemini Live API, enabling real-time input/output, barge-in, voice configuration, live transcription/translation, and tool calls during a spoken session. It differentiates when to use Live vs Live Translation modes and provides guidance on session setup, latency, and governance concerns.

How it works

Describes that Gemini Live is a stateful WebSocket API for low-latency interactions across audio, images, text, and potentially video, returning spoken output and potential function-call events. Specifies transport as a persistent WebSocket (BidiGenerateContent endpoint for raw WebSockets) and front-loaded session setup including model, modalities, parameters, system instructions, and tools. Outlines audio input as 16-bit PCM little-endian at 16 kHz; output as 16-bit PCM little-endian at 24 kHz; recommends 20-40 ms input chunks and a practical 100 ms client buffering cap. Explains handling of transcripts, voice selection via speechConfig.voiceConfig.prebuiltVoiceConfig.voiceName, and features like barge-in, VAD, and grounding with Google Search. Details tool-calling behavior: client must process tool-call events and return FunctionResponse objects; behavior varies by model (synchronous vs asynchronous). Provides cost, quotas, governance notes, including session limits, resumption, and data residency considerations. Advises design of architecture for latency, tool governance, and user consent. Includes sections on system instruction design, latency setup, and safety for tool calls, with explicit configurations and prompts to ensure compliant activity.

When to use it

Use for live spoken interactions requiring real-time turn-taking, interruptions, and potential tool calls within a session. Do not use Gemini Live when the deliverable is a finished voiceover, offline transcript, or prerecording; prefer TTS or STT for those cases unless live interaction is explicitly required. Use Live Translation mode for low-latency speech-to-speech interpretation without tools or multimodal reasoning.

What it can touch

References tools and behaviors: function calling, Google Search grounding, and a set of model-specific capabilities. It describes how to schema tool calls narrowly, including required enums, units, locale, timezone, irreversible/confirmation flags, and user-visible confirmations within tool responses. It also discusses grounding with Google Search and the associated pricing/data-retention implications.

Caveats

Notes Live API is generally documented as Preview for Developer API, with Enterprise Platform offering separate data-residency and governance controls. Highlights that session limits, tokenization, and data retention vary by surface and model; resumption and retention policies are surface-specific and must be verified before production. Mentions pricing details and the need to re-check pricing before quoting.