Handling media data

View as Markdown

Realtime Media Streams (RTMS) delivers audio and transcript data from Zoom Phone calls over WebSocket connections between your application and the RTMS servers.

After establishing a connection to the RTMS server, your application receives media data based on:

Zoom Phone streams provide two media types: audio (media_type 1) and transcript (media_type 8).

Audio

Audio data is available per participant. Each participant has their own audio packets, identified by a channel_id.

The RTMS server sends audio data packets from participants in base64-encoded binary format. The decoded data is raw PCM audio.

Each packet of data contains the participant's channel identifier (channel_id) and the timestamp (timestamp) of the audio data. Your app can use the channel_id to associate the audio with the participant metadata sent in the CALL_INFO_UPDATED event.

Example audio packet

{
    "msg_type": 14,
    "content": {
        "channel_id": "123abc",
        "data": "Hw1kDacNAA4sDkMOAQ5eDekMgAzXCw0L4Al4CDwHBAayBEwDmwHD/wn+rfyQ+3z6Z/k4+Ef3jvb09aj1YPUF9QL1OfVt9eH1f/YE96P3jPib+dX6Pvyr/ef+IABcAZ4=",
        "timestamp": 1738392033699
    }
}

Timestamps

Timestamps denote the creation time on Zoom's server. The timestamp for each audio packet changes based on the send interval of the stream.

When working with streaming audio, timestamps are useful in determining the sequence of messages. Use timestamps to infer the period of time where a participant might be muted.

Data processing

When processing audio data, a separate file is needed for each channel_id. Once you have each separate file, you can combine them into one file.

Find the channel ID

Before you can save the files, find and save the channel IDs. Add the following code to the audio WebSocket message handler:

ws.on('message', (data) => {
    const message = JSON.parse(data.toString());

    if (message.msg_type === 4 && message.status_code === 0) {
        // Media handshake succeeded: send CLIENT_READY_ACK to the signaling connection
        send(session.signalingWs, {
            msg_type: 7,
            rtms_stream_id: session.streamId,
        });
    } else if (message.msg_type === 12) {
        // Keep-alive request
        send(ws, { msg_type: 13, timestamp: message.timestamp });
    } else if (message.msg_type === 14) {
        // Audio data
        const audioBuffer = Buffer.from(message.content.data, 'base64');
        const channelId = message.content.channel_id;

        if (!session.channelPaths.has(channelId)) {
            const rawPath = getChannelRawPath(session.sessionDir, channelId);
            const wavPath = getChannelWavPath(session.sessionDir, channelId);
            session.channelPaths.set(channelId, { rawPath, wavPath });
            console.log(`New channel ${channelId} → ${rawPath}`);
        }

        const { rawPath } = session.channelPaths.get(channelId);
        saveRawAudio(audioBuffer, rawPath);
    }
});

Convert and save stream

Now that you have the channel IDs, convert the stream data, save it to a file, and then combine it into a single file, if you want.

The following helpers save the raw PCM data for each channel and convert it to WAV with FFmpeg:

import fs from 'fs';
import path from 'path';
import { exec } from 'child_process';
import { promisify } from 'util';

const execAsync = promisify(exec);

// Cache of open write streams keyed by file path
const writeStreams = new Map();

export function getChannelRawPath(sessionDir, channelId) {
    return path.join(sessionDir, `channel_${channelId}.raw`);
}

export function getChannelWavPath(sessionDir, channelId) {
    return path.join(sessionDir, `channel_${channelId}.wav`);
}

export function saveRawAudio(chunk, rawPath) {
    const dir = path.dirname(rawPath);
    if (!fs.existsSync(dir)) {
        fs.mkdirSync(dir, { recursive: true });
    }

    let stream = writeStreams.get(rawPath);
    if (!stream) {
        stream = fs.createWriteStream(rawPath, { flags: 'a' });
        writeStreams.set(rawPath, stream);
    }

    stream.write(chunk);
}

export async function convertRawToWav(inputFile, outputFile, options = {}) {
    // Confirm the sample rate of Zoom Phone audio before you use this default
    const sampleRate = options.sampleRate || 16000;
    const channels = options.channels || 1;
    const command = `ffmpeg -y -f s16le -ar ${sampleRate} -ac ${channels} -i "${inputFile}" "${outputFile}"`;
    try {
        await execAsync(command);
    } catch (error) {
        throw new Error(`FFmpeg conversion failed: ${error.message}`);
    }
}

export function closeRawStream(rawPath) {
    const stream = writeStreams.get(rawPath);
    if (!stream) return Promise.resolve();
    return new Promise((resolve) => {
        stream.end(() => {
            writeStreams.delete(rawPath);
            resolve();
        });
    });
}

Best practices

When a call starts, capture the first timestamp from the signaling connection. This denotes the start of the call.

When participants mute their microphones, the RTMS server stops sending audio packets for that participant. Use timestamps to detect these gaps and insert silence if your application needs it.

When working with pulse-coded modulation (PCM) audio, consider the following:

  • The size of raw PCM buffers and the storage they take up without compression.
  • You may need to convert the audio to WAV format to use services such as live streaming or speech to text transcription.
  • Compressing to lossy formats, such as mp3, requires the entire audio file to be complete before compression. We recommend you compress the audio after the call has ended.

Transcripts

Transcript data is available per participant with attribution, also called diarization.

Transcripts arrive continuously as speech is detected and processed in real time.

The RTMS server sends transcript data packets from each participant as text data with the participant's channel identifier (channel_id), the detected language, and the timestamp.

Example transcript packet

{
    "msg_type": 17,
    "content": {
        "channel_id": "c8b2d3ce-b0d6-4cd1-9195-fe17a4bd5b8e",
        "start_time": 1738392033699,
        "end_time": 1738392036866,
        "timestamp": 1727384349000000,
        "language": 9,
        "data": "Hi, hello world!"
    }
}

Languages

The language field provides the automatically detected language spoken. When a participant switches from one language to another, automatic detection takes a short time to identify the new language.

Timestamps

Timestamps are sent when the sentence or utterance begins. Use them to create a log of when a participant began speaking.

Best practices

Zoom's transcription optimizes for low latency, so it may help to post-process text transcripts into final assets.

In sentence detection, pauses in speech can sometimes be mistaken for the end of a sentence.

Combining multiple transcript messages using a message interval and timeouts is a useful strategy to combine disjointed sentences.