Receive raw data

You receive a remote user's raw video and share frames by subscribing to their pipe with a ZoomVideoSDKRawDataPipeDelegate, and raw audio through your ZoomVideoSDKDelegate callbacks or a virtual speaker.

Receive raw video data

Raw video data is offered in the following types:

• YUV420: A data object commonly used by the renderer based on OpenGL ES.
• CVPixelBuffer defined in NV12: A data object defined by Apple that can be used with a Metal renderer.

To access and modify this data, you must:

1. Implement an instance of the ZoomVideoSDKRawDataPipeDelegate.
2. Use the callback functions provided by the ZoomVideoSDKRawDataPipeDelegate to receive each frame of the raw video data.
3. Pass the delegate into the video pipe of a specific user.

Implement ZoomVideoSDKRawDataPipeDelegate

// Used to receive video's NV12 data (CVPixelBufferRef).
func onPixelBuffer(_ pixelBuffer: CVPixelBuffer!, rotation: ZoomVideoSDKVideoRawDataRotation) {
    // Access CVPixelBufferRef using pixelBuffer.
    // Get rotation of raw data video stream.
    switch rotation {
    case ZoomVideoSDKVideoRawDataRotation90:
        // 90 degrees.
        break
    default:
        break
    }
}
// Used to receive video's YUV420 data.
func onRawDataFrameReceived(_ rawData: ZoomVideoSDKVideoRawData!) {
    // Access the raw data for each of the 3 components — see "Read a video frame" below.
}
// Called when the sender stops or starts sending raw data.
func onRawDataStatusChanged(_ userRawdataStatus: ZoomVideoSDKUserRawdataStatus) {
    switch userRawdataStatus {
    case ZoomVideoSDKUserRawdataOn:
        // User's raw data is now on.
        break
    case ZoomVideoSDKUserRawdataOff:
        // User's raw data is now off.
        break
    default:
        break
    }
}

// Used to receive video's NV12 data (CVPixelBufferRef).
- (void)onPixelBuffer:(CVPixelBufferRef)pixelBuffer rotation:(ZoomVideoSDKVideoRawDataRotation)rotation {
    // Access CVPixelBufferRef using pixelBuffer.
    // Get rotation of raw data video stream.
    switch (rotation) {
        case ZoomVideoSDKVideoRawDataRotation90:
            // 90 degrees.
            break;
        default:
            break;
    }
}
// Used to receive video's YUV420 data.
- (void)onRawDataFrameReceived:(ZoomVideoSDKVideoRawData *)rawData {
    // Access the raw data for each of the 3 components — see "Read a video frame" below.
}
// Called when the sender stops or starts sending raw data.
- (void)onRawDataStatusChanged:(ZoomVideoSDKUserRawdataStatus)userRawdataStatus {
    switch (userRawdataStatus) {
        case ZoomVideoSDKUserRawdataOn:
            // User's raw data is now on.
            break;
        case ZoomVideoSDKUserRawdataOff:
            // User's raw data is now off.
            break;
    }
}

Note

It is not recommended to implement a callback that does not belong to the renderer you are using. It is also not recommended to implement both onRawDataFrameReceived and onPixelBuffer. Implementing both callbacks will cause unnecessary and multiple raw data manipulations.

To read the per-pixel alpha mask alongside the YUV frame, see Raw video with alpha channel.

Subscribe to a user's video pipe

To start receiving frames, pass your delegate into the video pipe of a specific user and check the return value.

guard let user = session.getMySelf(),
      let pipe = user.getVideoPipe() else { return }
let result = pipe.subscribe(with: delegate, resolution: ZoomVideoSDKVideoResolution_720)
if result != .Errors_Success {
    // Subscription was rejected.
}

ZoomVideoSDKUser *user = [session getMySelf];
ZoomVideoSDKError result = [user.getVideoPipe subscribeWithDelegate:delegate resolution:ZoomVideoSDKVideoResolution_720];
if (result != Errors_Success) {
    // Subscription was rejected.
}

subscribeWithDelegate:resolution: returns a ZoomVideoSDKError. Errors_Success means frames will start arriving on onRawDataFrameReceived (or onPixelBuffer); any other value is an error code and no frames are delivered. A concrete resolution is required — Auto is not accepted for raw-data subscriptions.

Read a video frame

Inside onRawDataFrameReceived, read the planes and metadata from the ZoomVideoSDKVideoRawData.

let size = rawData.size              // CGSize — frame width and height
let yBuffer = rawData.yBuffer        // char * for the Y plane
let uBuffer = rawData.uBuffer        // char * for the U plane
let vBuffer = rawData.vBuffer        // char * for the V plane
let rotation = rawData.rotation      // ZoomVideoSDKVideoRawDataRotationNone / ...90 / ...180 / ...270
let format = rawData.format          // ZoomVideoSDKFrameDataFormat_I420 or ..._I420_Limit
let timeStamp = rawData.timeStamp    // NSDate

CGSize size = [rawData size];                          // frame width and height
char *yBuffer = [rawData yBuffer];                      // Y plane
char *uBuffer = [rawData uBuffer];                      // U plane
char *vBuffer = [rawData vBuffer];                      // V plane
ZoomVideoSDKVideoRawDataRotation rotation = [rawData rotation];
ZoomVideoSDKFrameDataFormat format = [rawData format];
NSDate *timeStamp = [rawData timeStamp];
switch (format) {
    case ZoomVideoSDKFrameDataFormat_I420:
        // Raw data is I420 format.
        break;
    case ZoomVideoSDKFrameDataFormat_I420_Limit:
        // Raw data is I420 (limited range) format.
        break;
}

The frame's dimensions come from size (a CGSize); there are no separate streamWidth / streamHeight accessors on iOS.

Retain a frame past the callback

Raw data buffers are valid only inside onRawDataFrameReceived by default. Once the callback returns, the buffer may be recycled. To hold onto a frame — for example, to hand it off to a background thread for processing or encoding — increase its reference count with addRef, then release it with releaseRef when you are done.

func onRawDataFrameReceived(_ rawData: ZoomVideoSDKVideoRawData!) {
    guard rawData.canAddRef() else { return }
    rawData.addRef()
    backgroundQueue.async {
        defer { rawData.releaseRef() }
        self.processFrame(rawData)
    }
}

- (void)onRawDataFrameReceived:(ZoomVideoSDKVideoRawData *)rawData {
    if (![rawData canAddRef]) {
        return;
    }
    [rawData addRef];
    dispatch_async(self.backgroundQueue, ^{
        [self processFrame:rawData];
        [rawData releaseRef];
    });
}

Check canAddRef before calling addRef, and always pair addRef with releaseRef. Orphaned refs leak frame buffers.

Stop receiving frames

To stop receiving frames, call unSubscribeWithDelegate: on the same pipe with the delegate you subscribed. Always unsubscribe before tearing down the owning view controller, otherwise the SDK keeps invoking your delegate against a deallocated context.

user.getVideoPipe()?.unSubscribe(with: delegate)

[user.getVideoPipe unSubscribeWithDelegate:delegate];

Receive raw audio data

Through your implementation of ZoomVideoSDKDelegate, you can access mixed (combined audio output from one or more users in a session), per-user, and shared raw audio data. Receive audio from hardware devices or virtual audio if it was sent through ZoomVideoSDKVirtualAudioMic.

Unlike raw video data, raw audio data will default to stack-based memory if you do not specify a memory mode.

• The virtual speaker allows access to audio data received from other users in the session. This data represents what a user would hear played through the device's speakers.
• The virtual mic allows audio data for the current user to be sent to the session programmatically instead of from the SDK capturing it through an audio input device.

Once you have implemented the memory mode callbacks, you must subscribe to the listener to receive audio data through an instance of ZoomVideoSDKAudioHelper.

func onMixedAudioRawDataReceived(_ rawData: ZoomVideoSDKAudioRawData!) {
    // Access audio raw data for the whole session here.
}
func onSharedAudioRawDataReceived(_ rawData: ZoomVideoSDKAudioRawData!) {
    // Access share audio raw data here.
}
func onOneWayAudioRawDataReceived(_ rawData: ZoomVideoSDKAudioRawData!, user: ZoomVideoSDKUser!) {
    // Access per-user audio raw data here.
}

- (void)onMixedAudioRawDataReceived:(ZoomVideoSDKAudioRawData *)rawData {
    // Access audio raw data for the whole session here.
}
- (void)onSharedAudioRawDataReceived:(ZoomVideoSDKAudioRawData *)rawData {
    // Access share audio raw data here.
}
- (void)onOneWayAudioRawDataReceived:(ZoomVideoSDKAudioRawData *)rawData user:(ZoomVideoSDKUser *)user {
    // Access per-user audio raw data here.
}

Inside each callback, the ZoomVideoSDKAudioRawData exposes the PCM buffer through buffer (with bufferLen), along with sampleRate and channelNum.

Receive raw audio for virtual speaker

Use the virtual speaker to process audio sent to the speaker, and the virtual microphone to process audio received from the microphone. If you receive or send audio directly, you won't be able to process it.

Follow these steps to receive virtual audio.

1. Create an instance of ZoomVideoSDKVirtualAudioSpeaker.
2. Pass that instance into ZoomVideoSDKSessionContext.
3. Access raw data in each callback method.

VirtualSpeakerExample.swift

import Foundation
class VirtualSpeakerExample: NSObject, ZoomVideoSDKVirtualAudioSpeaker {
    func onVirtualSpeakerMixedAudioReceived(_ rawData: ZoomVideoSDKAudioRawData!) {
        // Received raw audio from the whole session that was sent from virtual microphones.
        // Play the raw audio here.
    }
    func onVirtualSpeakerOneWayAudioReceived(_ rawData: ZoomVideoSDKAudioRawData!, user: ZoomVideoSDKUser!) {
        // Received raw audio from a single user that was sent from a virtual microphone.
    }
    func onVirtualSpeakerSharedAudioReceived(_ rawData: ZoomVideoSDKAudioRawData!) {
        // Received raw audio from share that was manually sent.
    }
}

VirtualSpeakerExample.h

#import <Foundation/Foundation.h>
NS_ASSUME_NONNULL_BEGIN
@interface VirtualSpeakerExample : NSObject <ZoomVideoSDKVirtualAudioSpeaker>
@end
NS_ASSUME_NONNULL_END

VirtualSpeakerExample.m

#import "VirtualSpeakerExample.h"
@implementation VirtualSpeakerExample
- (void)onVirtualSpeakerMixedAudioReceived:(ZoomVideoSDKAudioRawData *)rawData {
    // Received raw audio from the whole session that was sent from virtual microphones.
    // Play the raw audio here.
}
- (void)onVirtualSpeakerOneWayAudioReceived:(ZoomVideoSDKAudioRawData *)rawData user:(ZoomVideoSDKUser *)user {
    // Received raw audio from a single user that was sent from a virtual microphone.
}
- (void)onVirtualSpeakerSharedAudioReceived:(ZoomVideoSDKAudioRawData *)rawData {
    // Received raw audio from share that was manually sent.
}
@end

Receive raw share data

Receive raw share video and audio data, for example when someone is sharing their screen, similarly to how you receive raw video and raw audio data.

Receive raw share video data

Follow the same steps and code to receive raw video data, except get the share pipe instead of the video pipe, as shown in the following code.

// Previous code same as receive raw video.
// Pass the delegate into the share pipe of a specific user (for example, myself).
guard let user = session.getMySelf(),
      let pipe = user.getSharePipe() else { return }
let result = pipe.subscribe(with: delegate, resolution: ZoomVideoSDKVideoResolution_720)
if result != .Errors_Success {
    // Subscription was rejected.
}

// Previous code same as receive raw video.
// Pass the ZoomVideoSDKRawDataPipeDelegate object into the share pipe of a specific user (for example, myself).
ZoomVideoSDKUser *user = [session getMySelf];
ZoomVideoSDKError result = [user.getSharePipe subscribeWithDelegate:delegate resolution:ZoomVideoSDKVideoResolution_720];
if (result != Errors_Success) {
    // Subscription was rejected.
}

The same onRawDataFrameReceived callback fires for share frames; the same accessors, reference-counting, and unsubscribe rules apply.

Receive raw share audio data

Receive raw share audio data in the same way as you receive raw audio data, through the onSharedAudioRawDataReceived callback on your ZoomVideoSDKDelegate, or onVirtualSpeakerSharedAudioReceived on a ZoomVideoSDKVirtualAudioSpeaker.