api-ai-elevenlabs

What it does

Guides using the ElevenLabs SDKs to perform text-to-speech (convert and stream), manage voices (search/get), clone voices, convert speech-to-speech, and build real-time conversational AI agents with voice interactions. It explains model choices (eleven_v3, eleven_flash_v2_5, eleven_multilingual_v2) and how to handle streams as ReadableStream<Uint8Array>, including piping to files or HTTP responses. It also covers pronunciation control, SSML, and error handling patterns.

How it works

• Use the official @elevenlabs/elevenlabs-js package for server-side TTS, voice management, and speech-to-speech, and @elevenlabs/client for real-time conversational AI agents.
• Initialize a client (examples show a minimal setup and production configuration) and rely on environment variable ELEVENLABS_API_KEY for authentication.
• For TTS, call textToSpeech.convert(VOICE_ID, { text, modelId, ... }) to generate full audio as a ReadableStream<Uint8Array>, or textToSpeech.stream(VOICE_ID, { text, modelId, ... }) for low-latency streaming; the SDK returns ReadableStream<Uint8Array> that must be consumed (pipe to file or HTTP response).
• Voice settings can be adjusted via voiceSettings (stability, similarityBoost, style, speed, etc.).
• Manage voices with voices.search() and voices.get(VOICE_ID). Voice cloning uses voices.ivc.create with sample files; speech-to-speech uses speechToSpeech.convert with an input audio stream and model.
• Examples illustrate consuming streams, piping to files, and error handling with ElevenLabsError types.

When to use it

• Generating speech audio from text (narration, audiobooks, announcements)
• Streaming audio in real-time for low-latency playback
• Cloning voices from audio samples
• Converting speech from one voice to another (speech-to-speech)
• Building real-time conversational AI agents with voice interaction
• Controlling pronunciation with SSML or pronunciation dictionaries
• Generating audio with character-level timestamp alignment

What it can touch

• Client initialization and configuration (retries, timeouts, API key)
• Text-to-speech conversion and streaming (convert, stream, timestamps)
• Voice settings (stability, similarityBoost, style, speed)
• Voice selection and management (voices.search, voices.get)
• Voice cloning (voices.ivc.create)
• Speech-to-speech (speechToSpeech.convert)
• WebSocket input streaming for real-time patterns
• Pronunciation dictionaries and SSML
• Conversational AI agents ( @elevenlabs/client )

Caveats

• Requires environment variable ELEVENLABS_API_KEY; never hardcode keys.
• Streams must be consumed to avoid resource leaks.
• voiceId must be passed as the first positional argument to textToSpeech methods (not inside options).
• Model selection should match use case (quality vs latency vs long-form stability).
• Handles specific error types (ElevenLabsTimeoutError, ElevenLabsError) for robust debugging.