There are two ways to consume incoming media as raw data:
- Video and share frames: subscribe a
ZoomVideoSDKRawDataPipeDelegateto a user's video or share pipe. - Audio frames: install a
ZoomVideoSDKVirtualAudioSpeakeron the session context at join time, or listen for the audio raw-data callbacks on yourZoomVideoSDKDelegate.
Receive raw video frames
Raw video arrives as YUV (I420) data: three planes (Y, U, V) plus dimensions and rotation. Implement ZoomVideoSDKRawDataPipeDelegate and pass it to a user's video pipe.
val dataDelegate = object : ZoomVideoSDKRawDataPipeDelegate {
override fun onRawDataStatusChanged(status: ZoomVideoSDKRawDataPipeDelegate.RawDataStatus?) {
// The pipe is now active or inactive.
}
override fun onRawDataFrameReceived(rawData: ZoomVideoSDKVideoRawData?) {
rawData ?: return
// Read YUV planes and dimensions — see "Read a video frame" below.
}
}
val pipe = user.videoPipe
val result = pipe.subscribe(
ZoomVideoSDKVideoResolution.VideoResolution_360P,
dataDelegate
)
if (result != ZoomVideoSDKErrors.Errors_Success) {
// Subscription was rejected.
}
ZoomVideoSDKRawDataPipeDelegate dataDelegate = new ZoomVideoSDKRawDataPipeDelegate() {
@Override
public void onRawDataStatusChanged(ZoomVideoSDKRawDataPipeDelegate.RawDataStatus status) {
// The pipe is now active or inactive.
}
@Override
public void onRawDataFrameReceived(ZoomVideoSDKVideoRawData rawData) {
if (rawData == null) return;
// Read YUV planes and dimensions — see "Read a video frame" below.
}
};
ZoomVideoSDKRawDataPipe pipe = user.getVideoPipe();
int result = pipe.subscribe(
ZoomVideoSDKVideoResolution.VideoResolution_360P,
dataDelegate
);
if (result != ZoomVideoSDKErrors.Errors_Success) {
// Subscription was rejected.
}
subscribe returns an int: ZoomVideoSDKErrors.Errors_Success (value 0) means frames will start arriving on onRawDataFrameReceived; any other value is an error code from ZoomVideoSDKErrors .
Read a video frame
Inside onRawDataFrameReceived, read planes and metadata from the ZoomVideoSDKVideoRawData.
val width = rawData.streamWidth
val height = rawData.streamHeight
val yBuffer = rawData.yBuffer // ByteBuffer for the Y plane
val uBuffer = rawData.uBuffer // ByteBuffer for the U plane
val vBuffer = rawData.vBuffer // ByteBuffer for the V plane
val rotation = rawData.rotation // 0, 90, 180, or 270 (clockwise)
val format = rawData.format // currently always FORMAT_TYPE_I420
val timestamp = rawData.timeStamp
int width = rawData.getStreamWidth();
int height = rawData.getStreamHeight();
ByteBuffer yBuffer = rawData.getyBuffer(); // Y plane
ByteBuffer uBuffer = rawData.getuBuffer(); // U plane
ByteBuffer vBuffer = rawData.getvBuffer(); // V plane
int rotation = rawData.getRotation(); // 0, 90, 180, or 270 (clockwise)
int format = rawData.getFormat(); // currently always FORMAT_TYPE_I420
long timestamp = rawData.getTimeStamp();
The getter casing (getyBuffer, getuBuffer, getvBuffer) reflects the SDK method names. Don't "correct" the lowercase letter when grepping or wiring up reflection.
Retain a frame past the callback
Raw data buffers are valid only inside onRawDataFrameReceived by default. Once the callback returns, the buffer may be recycled. To hold onto a frame (for example, to hand it off to a background thread for processing or encoding), use the SDK's reference counting.
override fun onRawDataFrameReceived(rawData: ZoomVideoSDKVideoRawData?) {
rawData ?: return
if (rawData.canAddRef()) {
rawData.addRef()
backgroundExecutor.execute {
try {
processFrame(rawData)
} finally {
rawData.releaseRef()
}
}
}
}
@Override
public void onRawDataFrameReceived(ZoomVideoSDKVideoRawData rawData) {
if (rawData == null) return;
if (rawData.canAddRef()) {
rawData.addRef();
backgroundExecutor.execute(() -> {
try {
processFrame(rawData);
} finally {
rawData.releaseRef();
}
});
}
}
Always pair addRef() with releaseRef(). Orphaned refs leak frame buffers.
Stop receiving frames
Call unSubscribe(delegate) on the same pipe when you no longer want frames. Always unsubscribe before tearing down the Activity, otherwise the SDK keeps invoking your delegate against a destroyed context.
user.videoPipe.unSubscribe(dataDelegate)
user.getVideoPipe().unSubscribe(dataDelegate);
Receive raw audio frames
Through your ZoomVideoSDKDelegate, you can listen for mixed (combined audio from one or more users), per-user, and share audio raw data callbacks. The callbacks fire continuously while any audio is flowing in the session.
override fun onMixedAudioRawDataReceived(rawData: ZoomVideoSDKAudioRawData?) {
// Combined audio for the whole session.
}
override fun onOneWayAudioRawDataReceived(
rawData: ZoomVideoSDKAudioRawData?,
user: ZoomVideoSDKUser
) {
// Per-user audio.
}
override fun onShareAudioRawDataReceived(rawData: ZoomVideoSDKAudioRawData?) {
// Audio from an active screen share.
}
@Override
public void onMixedAudioRawDataReceived(ZoomVideoSDKAudioRawData rawData) {
// Combined audio for the whole session.
}
@Override
public void onOneWayAudioRawDataReceived(
ZoomVideoSDKAudioRawData rawData,
ZoomVideoSDKUser user
) {
// Per-user audio.
}
@Override
public void onShareAudioRawDataReceived(ZoomVideoSDKAudioRawData rawData) {
// Audio from an active screen share.
}
Inside each callback, the ZoomVideoSDKAudioRawData exposes the PCM buffer through rawData.buffer (Kotlin) / rawData.getBuffer() (Java).
Receive audio through a virtual speaker
You can also install a ZoomVideoSDKVirtualAudioSpeaker on the session context when joining. This routes received audio to your speaker implementation instead of (or in addition to) the device's audio output. Use this when you want to process incoming audio without playing it through the speaker.
val virtualSpeaker = object : ZoomVideoSDKVirtualAudioSpeaker {
override fun onVirtualSpeakerMixedAudioReceived(rawData: ZoomVideoSDKAudioRawData?) {
// Combined session audio.
}
override fun onVirtualSpeakerOneWayAudioReceived(
rawData: ZoomVideoSDKAudioRawData?,
user: ZoomVideoSDKUser?
) {
// Per-user audio.
}
override fun onVirtualSpeakerShareAudioReceived(rawData: ZoomVideoSDKAudioRawData?) {
// Share audio.
}
}
val context = ZoomVideoSDKSessionContext().apply {
virtualAudioSpeaker = virtualSpeaker
}
ZoomVideoSDKVirtualAudioSpeaker virtualSpeaker = new ZoomVideoSDKVirtualAudioSpeaker() {
@Override
public void onVirtualSpeakerMixedAudioReceived(ZoomVideoSDKAudioRawData rawData) {
// Combined session audio.
}
@Override
public void onVirtualSpeakerOneWayAudioReceived(
ZoomVideoSDKAudioRawData rawData,
ZoomVideoSDKUser user
) {
// Per-user audio.
}
@Override
public void onVirtualSpeakerShareAudioReceived(ZoomVideoSDKAudioRawData rawData) {
// Share audio.
}
};
ZoomVideoSDKSessionContext context = new ZoomVideoSDKSessionContext();
context.virtualAudioSpeaker = virtualSpeaker;
Receive raw share data
A screen share streams as both video frames (the screen capture) and optionally audio (the device audio the sharer included with enableShareDeviceAudio).
Share video
Get the share pipe from the sharing user, then subscribe exactly like you would for video.
val pipe = user.sharePipe
pipe.subscribe(
ZoomVideoSDKVideoResolution.VideoResolution_360P,
dataDelegate
)
ZoomVideoSDKRawDataPipe pipe = user.getSharePipe();
pipe.subscribe(
ZoomVideoSDKVideoResolution.VideoResolution_360P,
dataDelegate
);
The same onRawDataFrameReceived callback fires for share frames; the same accessors and lifecycle rules apply.
Share audio
Listen for onShareAudioRawDataReceived on your ZoomVideoSDKDelegate (see Receive raw audio frames) or onVirtualSpeakerShareAudioReceived on a ZoomVideoSDKVirtualAudioSpeaker. The buffer format and access pattern is the same as session audio.