Handling transcript updates

View as Markdown

Realtime Media Streams (RTMS) sends a revised version of a transcript sentence when speech recognition changes it. The revision uses the same sentence_id as the original.

RTMS turns on transcript updates by default on the server. Your app doesn't need to request them. It only needs to handle the attribute value on each transcript message.

If language identification (enable_lid) is turned off, transcript updates are also turned off.

To receive source transcript data, include the transcript parameters in your media handshake request.

{
    "media_params": {
        "transcript": {
            "content_type": 5,
            "src_language": 9,
            "enable_lid": true
        }
    }
}

Set content_type to 5 (TEXT). src_language is a transcript language ID, such as 9 for English or 4 for Simplified Chinese. For more information, see Media parameter definitions.

Transcript message fields

Transcript messages have a top-level msg_type of 17, and the transcript payload is in content. This message is a new sentence from a participant in a Video SDK session.

{
    "msg_type": 17,
    "content": {
        "user_id": 42,
        "user_name": "Ada",
        "attribute": 1,
        "language": 9,
        "data": "We will start the review now.",
        "timestamp": 1730000000123,
        "start_time": 1730000000000,
        "end_time": 1730000001800,
        "sentence_id": 101
    }
}

The fields that matter for updates are:

FieldRequiredDescription
user_idYesThe speaker's identifier.
user_nameNoThe speaker's display name.
attributeYes1 for a new sentence, 2 for an update that replaces the sentence. The server doesn't send DELETE.
languageYesThe language of the returned text. It can differ from the requested source language when language identification is on.
dataYesThe full current text of the sentence, not a change to the previous text. Without payload encryption, it's UTF-8 text.
timestampYesWhen RTMS generated the transcript result, in milliseconds.
start_time and end_timeYesThe audio time range for the sentence, in milliseconds.
sentence_idYes for updatesThe ID of the sentence. 0 is a valid ID, so don't treat it as missing.

If payload encryption is on, data contains Base64-encoded encrypted bytes. Decrypt it before you read it as text. For how payload encryption is negotiated, see Signaling handshake request. The rules for attribute and sentence_id don't change.

Process transcript updates

Identify each sentence with both the speaker and the sentence_id.

(user_id, sentence_id)

Don't use sentence_id alone, because it isn't unique across speakers. Don't match updates by timestamp or arrival order.

  1. For attribute: 1, create the sentence with this key. If the sentence already exists, overwrite it.
  2. For attribute: 2, replace the sentence's text with the full data value. Also update language, start_time, end_time, and timestamp.
  3. If an update arrives for a sentence your app doesn't have, for example after a reconnect, create the sentence from the update and log the mismatch. Don't discard the update.
  4. Process every message idempotently. Don't add an update as a second sentence, and don't append its text to the existing sentence.
  5. A sentence can receive several updates. Keep the most recent version. Use audio timestamps to order sentences for display, but don't ignore a later update because of its timestamp.

Examples

The examples below show common cases. Timestamps are illustrative.

New transcript

RTMS sends a new sentence with attribute set to 1. Create the sentence with the key (42, 101).

{
    "msg_type": 17,
    "content": {
        "user_id": 42,
        "user_name": "Ada",
        "attribute": 1,
        "language": 9,
        "data": "We will start the review now.",
        "timestamp": 1730000000123,
        "start_time": 1730000000000,
        "end_time": 1730000001800,
        "sentence_id": 101
    }
}

Update to an existing sentence

Speech recognition first returns text that's incomplete or inaccurate, then rewrites the same sentence_id. The update contains the full replacement text. Replace the sentence at (42, 101) and update its time range.

{
    "msg_type": 17,
    "content": {
        "user_id": 42,
        "user_name": "Ada",
        "attribute": 2,
        "language": 9,
        "data": "We will start the design review now.",
        "timestamp": 1730000001450,
        "start_time": 1730000000000,
        "end_time": 1730000002200,
        "sentence_id": 101
    }
}

Several updates to the same sentence

The same sentence can be rewritten more than once. Keep only the latest version at (42, 101).

{
    "msg_type": 17,
    "content": {
        "user_id": 42,
        "attribute": 2,
        "language": 9,
        "data": "We will start the detailed design review now.",
        "timestamp": 1730000001960,
        "start_time": 1730000000000,
        "end_time": 1730000002600,
        "sentence_id": 101
    }
}

Update without an existing sentence

This can happen if your app reconnects after RTMS sent the new sentence. Store the update as the current version of (42, 205). Log that it was recovered from an update so you can find gaps in the data later.

{
    "msg_type": 17,
    "content": {
        "user_id": 42,
        "attribute": 2,
        "language": 4,
        "data": "现在开始设计评审。",
        "timestamp": 1730000010123,
        "start_time": 1730000009000,
        "end_time": 1730000011000,
        "sentence_id": 205
    }
}