Text-to-Speech

For an end-to-end spoken conversation, start with the Voice quickstart. This page covers individual speech providers and audio-generation tools.

OpenAI speech providers

Set OPENAI_API_KEY on the server and install timbal[server,voice]:

from timbal import Agent

agent = Agent(
    name="VoiceAgent",
    model="openai/gpt-5-mini",
    voice_config={
        "stt_provider": "openai",
        "stt_model": "gpt-transcribe",
        "tts_provider": "openai",
        "tts_model": "gpt-4o-mini-tts",
        "voice": "coral",
        "language": "en",
    },
)

These are independent STT/TTS providers in the regular Timbal voice pipeline: the agent still chooses its own LLM and tools. You can mix OpenAI transcription with ElevenLabs or Deepgram speech output, or select OpenAI only for TTS. WebSocket, WebRTC, LiveKit and telephony sessions use the same provider wiring. Both selectors are available in the voice playground; the advanced STT model field accepts an explicit transcription model. OpenAIRealtimeSTT uses a transcription-only Realtime WebSocket session. The default is gpt-transcribe. Legacy gpt-4o-transcribe, gpt-4o-mini-transcribe (including its 2025-03-20 and 2025-12-15 snapshots), and whisper-1 are also accepted. This adapter uses server VAD, keeping Timbal’s turn detector active. It supports local force-commit, accumulates partial transcripts, and delivers completed transcripts in audio-turn order. gpt-live-transcribe and gpt-realtime-whisper need client-side VAD and are explicitly rejected, as are diarization models. Legacy transcription models are deprecated; new integrations should use the default. STT tuning lives in stt_extra: prompt, threshold (default 0.5), prefix_padding_ms (300), silence_duration_ms (500), and noise_reduction ("near_field", "far_field", or None). The session’s language is an optional ISO-639-1 transcription hint. OpenAIStreamTTS uses the Speech API with streaming PCM audio. It defaults to gpt-4o-mini-tts and coral; tts-1 and tts-1-hd are also supported. Set voice to a built-in voice such as alloy, coral, marin, or cedar (tts-1 / tts-1-hd support only alloy, ash, coral, echo, fable, onyx, nova, sage, and shimmer). Invalid OpenAI model ids and unsupported model/voice combinations fail before opening a request. OPENAI_TTS_VOICE sets the fallback voice when a foreign provider’s voice id is inherited. Foreign model ids are replaced with the OpenAI defaults. Use tts_extra={"speed": 1.1, "instructions": "Speak warmly and clearly."} for speech style; instructions applies only to gpt-4o-mini-tts. The speech endpoint accepts complete text segments, so Timbal synthesizes each segment as it becomes ready rather than opening an incremental text context. Interrupting a segment closes its response; subsequent segments have fresh audio/resampling state. Each segment makes a separate HTTP request; this can introduce pauses and has different latency from ElevenLabs incremental text streaming. Benchmark with your own call audio before choosing a model. OpenAI’s wire audio is 24 kHz PCM16. The providers resample in both directions to the configured session sample rate, including the default 16 kHz. Rates other than 24 kHz require the timbal[voice] extra. PCM framing and resampler tails are preserved so playback and recordings use the advertised rate. Server-only stt_host / tts_host extras can select an API-compatible endpoint; clients cannot override the host. Credentials stay on the server. For file transcription or standalone audio generation, use the built-in tools:

fragment

import asyncio

from timbal.tools import OpenAISpeechToText, OpenAITextToSpeech


async def main() -> None:
    # Needs OPENAI_API_KEY and a local audio file.
    transcript = await OpenAISpeechToText()(
        audio_file_path="recording.wav",
        language="en",
    ).collect()
    audio = await OpenAITextToSpeech()(
        text="Hello from Timbal!",
        voice="coral",
        response_format="mp3",
    ).collect()
    print(transcript.output, audio.output)


asyncio.run(main())

Both tools also work in Agent(tools=[...]) and accept explicit api_key, platform integration, or OPENAI_API_KEY. OpenAISpeechToText accepts exactly one local path or base64 source (with filename for base64), returning JSON with text. OpenAITextToSpeech returns audio_base64, content_type, model, voice, and response_format. Formats: MP3, Opus, AAC, FLAC, WAV or raw PCM. Callers should identify generated speech as AI-generated in their user experience. Protocol references: Realtime transcription, Realtime session events, and text to speech.

Deepgram Aura

Set DEEPGRAM_API_KEY on the server. For live conversations, install timbal[server,voice] and select Deepgram in your agent’s voice configuration:

from timbal import Agent

agent = Agent(
    name="VoiceAgent",
    model="openai/gpt-5-mini",
    voice_config={
        "stt_provider": "deepgram-flux",
        "tts_provider": "deepgram",
        "tts_model": "aura-2-thalia-en",
    },
)

The same configuration works through the voice server’s WebSocket, WebRTC, LiveKit, and telephony transports. STT and TTS are independent: you can keep ElevenLabs transcription and use Deepgram for speech output. The voice playground also includes a Deepgram Aura provider option. Aura combines the voice and language into its model id. tts_model takes precedence; a full Aura id in voice is also accepted when tts_model contains another provider’s default. Otherwise, DEEPGRAM_TTS_MODEL is used, falling back to aura-2-thalia-en. Select the appropriate language’s voice explicitly; the language setting controls transcription and does not change an Aura voice. See the voice catalog. Live output is raw PCM16 at the session’s sample rate (8000, 16000, 24000, 32000, or 48000 Hz). Each reply streams text over its own connection and flushes once at the end. Interrupting a reply closes its socket and drops its queued audio. Greeting, filler, and reply streams are isolated. Each new reply pays a WebSocket handshake; no Deepgram SDK is required. This adapter uses Aura’s /v1/speak protocol, not the separate Flux TTS /v2/speak API. Optional server configuration: tts_extra={"speed": 1.1} adjusts speaking rate where supported. tts_extra also accepts mip_opt_out and a server-only tts_host for regional/self-hosted endpoints (for example, api.eu.deepgram.com). Client overrides cannot change the host or opt-out policy. For standalone audio generation and voice discovery, add the built-in tools:

from timbal import Agent
from timbal.tools import DeepgramListVoices, DeepgramTextToSpeech

agent = Agent(
    name="Narrator",
    model="openai/gpt-5-mini",
    tools=[DeepgramListVoices(), DeepgramTextToSpeech()],
)

Calling the tools directly needs DEEPGRAM_API_KEY:

fragment

import asyncio

from timbal.tools import DeepgramListVoices, DeepgramTextToSpeech


async def main() -> None:
    voices = await DeepgramListVoices()(language="es").collect()
    audio = await DeepgramTextToSpeech()(
        text="Hello from Timbal!",
        model="aura-2-thalia-en",
        encoding="mp3",
    ).collect()
    print(voices.output, audio.output)


asyncio.run(main())

DeepgramListVoices returns Aura model ids, languages and provider metadata (including voice sample URLs). DeepgramTextToSpeech returns audio_base64, content_type, model, and encoding. REST generation supports MP3, linear16, mu-law, A-law, Opus, FLAC and AAC; only pass sample rate, bitrate and container options supported by the selected format. Both tools support explicit api_key, platform integration, or DEEPGRAM_API_KEY credential resolution, following the same precedence as the ElevenLabs tools. For a custom voice pipeline, import DeepgramStreamTTS from timbal.voice and pass it as the tts provider to VoiceSession.

Custom OpenAI tool

Agents can also generate audio using a custom text-to-speech tool:

import os
from openai import AsyncOpenAI
from pydantic import Field
from timbal import Agent
from timbal.state import get_run_context
from timbal.types.file import File

async def tts(
    text: str = Field(
        ...,
        description="The text to convert to speech.",
    ),
) -> File:
    client = AsyncOpenAI(api_key=os.getenv("OPENAI_API_KEY"))
    
    kwargs = {
        "input": text,
        'model': 'tts-1',
        'voice': 'alloy'
    }
    
    response = await client.audio.speech.create(**kwargs)
    
    return File.validate(
        response.content,
        {"extension": ".mp3"}
    )

async def post_hook():
    """Post hook to extract and return the audio file."""
    span = get_run_context().current_span()
    span.output = await tts(text=span.output.collect_text())
    span.output.to_disk("audio.mp3")

agent = Agent(
    name="VoiceAgent",
    model="openai/gpt-5-mini",
    post_hook=post_hook,
)

# With OPENAI_API_KEY set, `await agent(prompt="What is the capital of France?").collect()`
# writes audio.mp3 from the reply. The hook calls the speech API, so it is not run here.

Key Features

Multiple Voices: Choose from various voice options
Audio Formats: Support for MP3, WAV, and other formats
File Output: Returns audio files for download or playback

Examples

Guides

Agents

Workflows

Text-to-Speech

OpenAI speech providers

Deepgram Aura

Custom OpenAI tool

Key Features

​OpenAI speech providers

​Deepgram Aura

​Custom OpenAI tool

​Key Features

OpenAI speech providers

Deepgram Aura

Custom OpenAI tool

Key Features