Handling transcript updates

View as Markdown

Realtime Media Streams (RTMS) sends a revised version of a transcript sentence when speech recognition changes it. The revision uses the same sentence_id as the original.

RTMS turns on transcript updates by default on the server. Your app doesn't need to request them. It only needs to handle the attribute value on each transcript message.

Open a transcript media connection with media_type set to 8 in your media handshake request. For more information, see Adding RTMS to your app.

Transcript message fields

Transcript messages have a top-level msg_type of 17, and the transcript payload is in content. This message is a new sentence from a Zoom Phone call.

{
    "msg_type": 17,
    "content": {
        "channel_id": "channel-9f4b",
        "attribute": 1,
        "language": 9,
        "data": "Thanks for calling. How can I help you today?",
        "timestamp": 1730000028100,
        "start_time": 1730000028000,
        "end_time": 1730000029500,
        "sentence_id": 18
    }
}

The fields that matter for updates are:

FieldRequiredDescription
channel_idYesThe channel identifier of the participant who spoke.
attributeYes1 for a new sentence, 2 for an update that replaces the sentence. The server doesn't send DELETE.
languageYesThe language of the returned text. It's the detected language of the speech.
dataYesThe full current text of the sentence, not a change to the previous text. Without payload encryption, it's UTF-8 text.
timestampYesWhen RTMS generated the transcript result, in milliseconds.
start_time and end_timeYesThe audio time range for the sentence, in milliseconds.
sentence_idYes for updatesThe ID of the sentence. 0 is a valid ID, so don't treat it as missing.

If payload encryption is on, data contains Base64-encoded encrypted bytes. Decrypt it before you read it as text. For how payload encryption is negotiated, see Signaling handshake request. The rules for attribute and sentence_id don't change.

Process transcript updates

Identify each sentence with both the channel and the sentence_id.

(channel_id, sentence_id)

Don't use sentence_id alone, because it isn't unique across channels. Don't match updates by timestamp or arrival order.

  1. For attribute: 1, create the sentence with this key. If the sentence already exists, overwrite it.
  2. For attribute: 2, replace the sentence's text with the full data value. Also update language, start_time, end_time, and timestamp.
  3. If an update arrives for a sentence your app doesn't have, for example after a reconnect, create the sentence from the update and log the mismatch. Don't discard the update.
  4. Process every message idempotently. Don't add an update as a second sentence, and don't append its text to the existing sentence.
  5. A sentence can receive several updates. Keep the most recent version. Use audio timestamps to order sentences for display, but don't ignore a later update because of its timestamp.

Examples

The examples below show common cases. Timestamps are illustrative.

New transcript

RTMS sends a new sentence with attribute set to 1. Create the sentence with the key (channel-9f4b, 18).

{
    "msg_type": 17,
    "content": {
        "channel_id": "channel-9f4b",
        "attribute": 1,
        "language": 9,
        "data": "Thanks for calling. How can I help you today?",
        "timestamp": 1730000028100,
        "start_time": 1730000028000,
        "end_time": 1730000029500,
        "sentence_id": 18
    }
}

Update to an existing sentence

Speech recognition first returns text that's incomplete or inaccurate, then rewrites the same sentence_id. The update contains the full replacement text. Replace the sentence at (channel-9f4b, 18) and update its time range.

{
    "msg_type": 17,
    "content": {
        "channel_id": "channel-9f4b",
        "attribute": 2,
        "language": 9,
        "data": "Your support case has been updated.",
        "timestamp": 1730000030200,
        "start_time": 1730000028000,
        "end_time": 1730000030000,
        "sentence_id": 18
    }
}

Several updates to the same sentence

The same sentence can be rewritten more than once. Keep only the latest version at (channel-9f4b, 18).

{
    "msg_type": 17,
    "content": {
        "channel_id": "channel-9f4b",
        "attribute": 2,
        "language": 9,
        "data": "Your support case has been updated and assigned.",
        "timestamp": 1730000031000,
        "start_time": 1730000028000,
        "end_time": 1730000031200,
        "sentence_id": 18
    }
}

Update without an existing sentence

This can happen if your app reconnects after RTMS sent the new sentence. Store the update as the current version of (channel-9f4b, 19). Log that it was recovered from an update so you can find gaps in the data later.

{
    "msg_type": 17,
    "content": {
        "channel_id": "channel-9f4b",
        "attribute": 2,
        "language": 9,
        "data": "Let me check that for you.",
        "timestamp": 1730000040123,
        "start_time": 1730000039000,
        "end_time": 1730000040500,
        "sentence_id": 19
    }
}