Live sessions

Voice Agent live sessions use JSON text frames over WebSocket. Input and output audio is Base64-encoded PCM16 inside JSON events.

Connect#

Open a WebSocket connection to:

wss://api.zoom.us/v2/ws/aiservices/voice-agent/live

Include the voice-agent.zoom.v2 subprotocol and your Zoom AI Services credentials in the upgrade request. The example sends a token in the Authorization: Bearer header.

import WebSocket from 'ws';

const socket = new WebSocket(
    'wss://api.zoom.us/v2/ws/aiservices/voice-agent/live',
    ['voice-agent.zoom.v2'],
    {
        headers: {
            Authorization: `Bearer ${process.env.ZOOM_ACCESS_TOKEN}`,
        },
    },
);

Configure the session#

The server sends session.created after the connection opens. Your first client event must be session.update. Session settings are grouped into three objects:

Send an empty object to use the defaults for that group:

{
    "type": "session.update",
    "session": {
        "listen": {},
        "think": {},
        "speak": {}
    }
}

This example sets each group explicitly:

{
    "type": "session.update",
    "session": {
        "listen": {
            "transcription": {
                "language": "en",
                "vocabulary": {
                    "phrases": ["Zoom", "Voice Agent"]
                }
            },
            "format": {
                "type": "pcm16",
                "rate": 16000
            }
        },
        "think": {
            "provider": {
                "type": "zoom",
                "model": "gemma-4"
            },
            "instructions": "You are a concise and friendly customer service representative.",
            "response_token_limit": "inf",
            "tools": [],
            "tool_choice": "auto",
            "reasoning": {
                "effort": "low"
            }
        },
        "speak": {
            "format": {
                "type": "pcm16"
            },
            "voice": "jenna"
        }
    }
}

The server confirms the effective configuration with session.updated. The following example shows the supplied instructions, defaults for omitted settings, and the session id:

{
    "type": "session.updated",
    "session": {
        "id": "43d2983171fb46af9c6c315e23ae65b9",
        "listen": {
            "transcription": {
                "language": "en"
            },
            "format": {
                "type": "pcm16",
                "rate": 16000
            }
        },
        "think": {
            "provider": {
                "type": "zoom",
                "model": "gemma-4"
            },
            "instructions": "You are a concise and friendly customer service representative.",
            "response_token_limit": "inf",
            "tool_choice": "auto"
        },
        "speak": {
            "format": {
                "type": "pcm16",
                "rate": 16000
            },
            "voice": "jenna"
        }
    }
}

A session lasts up to 60 minutes.

A successful session.update returns session.updated, which doesn't echo think.tools. Keep the tool definitions in your application.

Listen settings#

Think settings#

Speak settings#

Turn detection, noise reduction, speech speed, truncation, barge-in, and conversation history trimming use built-in defaults and aren't configurable.

Bring your own key#

To use your own OpenAI or Anthropic account, set think.provider.type to openai or anthropic, set model to a model from that provider, and pass your provider's API key in think.provider.headers. This is your OpenAI or Anthropic key, not your Zoom AI Services credentials:

{
    "type": "session.update",
    "session": {
        "listen": {},
        "think": {
            "provider": {
                "type": "openai",
                "model": "<provider-model-id>",
                "headers": {
                    "X-Api-Key": "<your-provider-api-key>"
                },
                "metadata": {
                    "context_window": 1000000,
                    "max_output_tokens": 128000
                }
            },
            "instructions": "You are a customer service representative."
        },
        "speak": {}
    }
}

headers is required when you use your own key. metadata describes the model's context window and maximum output tokens. session.updated never returns headers.

Stream input audio#

Send audio continuously, including silence, in input_audio_buffer.append events. The audio must match listen.format.

{
    "type": "input_audio_buffer.append",
    "audio": "<base64-encoded audio>"
}

Start another response#

Only one response can be active at a time. Before sending response.create for another response, wait for response.done for the active response. If you send response.create while a response is still active, the server returns a nonfatal error indicating that there is an active response. Wait for response.done, then retry.

Receiving response.output_audio.done or response.function_call_arguments.done does not mean the response has ended. Use response.done to determine when another response can start.

Handle server events#

Voice Agent emits events as input is detected and responses are generated.

Server events are wrapped in an envelope with a type and a payload object. payload contains the complete event, including its type, so read the event from payload. The event examples on this page show the event inside payload.

socket.on('message', (data) => {
    const envelope = JSON.parse(data.toString());
    const event = envelope.payload ?? envelope;

    if (event.type === 'response.output_audio.delta') {
        playPcm16(Buffer.from(event.delta, 'base64'));
    }

    if (event.type === 'error') {
        console.error(event.error);
    }
});

Cancel or truncate a response#

Use the server-generated IDs from response events:

• response.created.response.id identifies the response. Pass it as response_id in response.cancel.
• response.output_item.added.item.id identifies an output item. Audio events also include item_id; use the ID of the item whose audio you are truncating in conversation.item.truncate.

When a user interrupts audio that is already playing:

1. Send response.cancel to stop generation.
2. Stop local playback and record the last audio offset the user heard.
3. Send conversation.item.truncate so the conversation history matches the played audio.

{
    "type": "response.cancel",
    "response_id": "resp_abc123"
}

{
    "type": "conversation.item.truncate",
    "item_id": "item_abc123",
    "content_index": 0,
    "audio_end_ms": 1500
}

Use function calling#

Add function definitions to think.tools, and set think.tool_choice to auto (the model decides), none, or required (call at least one tool).

{
    "type": "session.update",
    "session": {
        "listen": {},
        "think": {
            "tools": [
                {
                    "type": "function",
                    "name": "get_order_status",
                    "description": "Get the current status of an order.",
                    "parameters": {
                        "type": "object",
                        "properties": {
                            "order_id": {
                                "type": "string"
                            }
                        },
                        "required": ["order_id"]
                    }
                }
            ],
            "tool_choice": "auto"
        },
        "speak": {}
    }
}

The server sends response.function_call_arguments.done when the function's arguments are complete:

{
    "type": "response.function_call_arguments.done",
    "event_id": "event_abc123",
    "response_id": "resp_abc123",
    "item_id": "item_abc123",
    "output_index": 0,
    "call_id": "call_abc123",
    "name": "get_order_status",
    "arguments": "{\"order_id\":\"123\"}"
}

Parse arguments as JSON and execute the named function in your application. Return its result as a function_call_output item using the event's call_id. The call_id links the result to the function call; it is not the response or item ID.

{
    "type": "conversation.item.create",
    "item": {
        "type": "function_call_output",
        "call_id": "call_abc123",
        "output": "{\"status\":\"shipped\",\"estimated_delivery\":\"Friday\"}"
    }
}

Wait for response.done for the response that requested the function before sending response.create to request the agent's next response.

{
    "type": "response.create"
}

Available voices#

Set speak.voice to one of these ten voice IDs. The default is jenna.

Handle errors#

The error.type field is invalid_request_error for recoverable client errors or server_error for service failures. The error.message field describes the failure.

{
    "type": "error",
    "event_id": "event_abc123",
    "error": {
        "type": "invalid_request_error",
        "message": "<description of the invalid setting>"
    }
}

When configuration validation fails, the server sends an error event. Unknown voices and unsupported input formats produce validation errors. Check session.updated for the effective settings. A rejected session.update does not disconnect the client. Correct the configuration and send session.update again on the same connection; a successful update returns session.updated.

An error indicating that there is an active response is also nonfatal. Wait for that response's response.done before retrying response.create.