Realtime Media Streams (RTMS) sends a revised version of a transcript sentence when speech recognition changes it. The revision uses the same sentence_id as the original.
RTMS turns on transcript updates by default on the server. Your app doesn't need to request them. It only needs to handle the attribute value on each transcript message.
If language identification (enable_lid) is turned off, transcript updates are also turned off.
To receive source transcript data, include the transcript parameters in your media handshake request.
{
"media_params": {
"transcript": {
"content_type": 5,
"src_language": 9,
"enable_lid": true
}
}
}
Set content_type to 5 (TEXT). src_language is a transcript language ID, such as 9 for English or 4 for Simplified Chinese. For more information, see Media parameter definitions.
Transcript message fields
Transcript messages have a top-level msg_type of 17, and the transcript payload is in content. This message is a new sentence from a participant in a meeting.
{
"msg_type": 17,
"content": {
"user_id": 42,
"user_name": "Ada",
"attribute": 1,
"language": 9,
"data": "We will start the review now.",
"timestamp": 1730000000123,
"start_time": 1730000000000,
"end_time": 1730000001800,
"sentence_id": 101
}
}
The fields that matter for updates are:
| Field | Required | Description |
|---|---|---|
user_id | Yes | The speaker's identifier. |
user_name | No | The speaker's display name. |
attribute | Yes | 1 for a new sentence, 2 for an update that replaces the sentence. The server doesn't send DELETE. |
language | Yes | The language of the returned text. It can differ from the requested source language when language identification is on. |
data | Yes | The full current text of the sentence, not a change to the previous text. Without payload encryption, it's UTF-8 text. |
timestamp | Yes | When RTMS generated the transcript result, in milliseconds. |
start_time and end_time | Yes | The audio time range for the sentence, in milliseconds. |
sentence_id | Yes for updates | The ID of the sentence. 0 is a valid ID, so don't treat it as missing. |
If payload encryption is on, data contains Base64-encoded encrypted bytes. Decrypt it before you read it as text. For how payload encryption is negotiated, see Signaling handshake request. The rules for attribute and sentence_id don't change.
Process transcript updates
Identify each sentence with both the speaker and the sentence_id.
(user_id, sentence_id)
Don't use sentence_id alone, because it isn't unique across speakers. Don't match updates by timestamp or arrival order.
- For
attribute: 1, create the sentence with this key. If the sentence already exists, overwrite it. - For
attribute: 2, replace the sentence's text with the fulldatavalue. Also updatelanguage,start_time,end_time, andtimestamp. - If an update arrives for a sentence your app doesn't have, for example after a reconnect, create the sentence from the update and log the mismatch. Don't discard the update.
- Process every message idempotently. Don't add an update as a second sentence, and don't append its text to the existing sentence.
- A sentence can receive several updates. Keep the most recent version. Use audio timestamps to order sentences for display, but don't ignore a later update because of its timestamp.
Examples
The examples below show common cases. Timestamps are illustrative.
New transcript
RTMS sends a new sentence with attribute set to 1. Create the sentence with the key (42, 101).
{
"msg_type": 17,
"content": {
"user_id": 42,
"user_name": "Ada",
"attribute": 1,
"language": 9,
"data": "We will start the review now.",
"timestamp": 1730000000123,
"start_time": 1730000000000,
"end_time": 1730000001800,
"sentence_id": 101
}
}
Update to an existing sentence
Speech recognition first returns text that's incomplete or inaccurate, then rewrites the same sentence_id. The update contains the full replacement text. Replace the sentence at (42, 101) and update its time range.
{
"msg_type": 17,
"content": {
"user_id": 42,
"user_name": "Ada",
"attribute": 2,
"language": 9,
"data": "We will start the design review now.",
"timestamp": 1730000001450,
"start_time": 1730000000000,
"end_time": 1730000002200,
"sentence_id": 101
}
}
Several updates to the same sentence
The same sentence can be rewritten more than once. Keep only the latest version at (42, 101).
{
"msg_type": 17,
"content": {
"user_id": 42,
"attribute": 2,
"language": 9,
"data": "We will start the detailed design review now.",
"timestamp": 1730000001960,
"start_time": 1730000000000,
"end_time": 1730000002600,
"sentence_id": 101
}
}
Update without an existing sentence
This can happen if your app reconnects after RTMS sent the new sentence. Store the update as the current version of (42, 205). Log that it was recovered from an update so you can find gaps in the data later.
{
"msg_type": 17,
"content": {
"user_id": 42,
"attribute": 2,
"language": 4,
"data": "现在开始设计评审。",
"timestamp": 1730000010123,
"start_time": 1730000009000,
"end_time": 1730000011000,
"sentence_id": 205
}
}
Updates with diarization
With diarization on, RTMS splits a sentence into parts for each speaker. Each part uses the speaker's user_id. The same sentence_id can appear for different speakers, so the speaker must be part of the key. This update changes (77, 310) only, and doesn't change (42, 310).
{
"msg_type": 17,
"content": {
"user_id": 77,
"attribute": 2,
"language": 9,
"data": "I agree with that plan.",
"timestamp": 1730000020500,
"start_time": 1730000020000,
"end_time": 1730000021800,
"sentence_id": 310
}
}