Raw data

The Video SDK provides you with an option to access real-time raw audio, video, and share data during a session, so you can apply additional effects and enhance the user experience. Use a processor component to apply effects or transformations to each piece of data in real-time.

An example of raw share data is when a user shares their screen.

  • Receive and send raw share video data using the same logic as video data, except get the share pipe instead of the video pipe in the raw data pipe method.
  • Receive and send raw share audio data using the same method used for raw audio data.

Follow the process outlined in this flowchart to capture and process raw data.

Flowchart outlining the raw data flow

Select raw data memory mode

In order to utilize raw data of any type, you must first select the memory mode you wish to use. The SDK supports heap-based and stack-based memory modes.

Stack-based memory

  • Variables are allocated automatically and deallocated after the data leaves scope.
  • Variables are not accessible from or transferable to other threads.
  • Typically has faster access than heap-based memory allocation.
  • Memory space is managed by the CPU and will not become fragmented.
  • Variables cannot be resized.

See Stack-based memory allocation for details.

Specify with:

ZoomVideoSDKRawDataMemoryMode.ZoomVideoSDKRawDataMemoryModeStack

Heap-based memory

  • Variables are allocated and deallocated manually. You must allocate and free variables to avoid memory leaks.
  • Variables can be accessed globally.
  • Has relatively slower access than stack-based memory allocation.
  • Has no guarantee on the efficiency of memory space and can become fragmented.
  • Variables can be resized.

See Heap-based dynamic memory allocation for details.

Specify with:

ZoomVideoSDKRawDataMemoryMode.ZoomVideoSDKRawDataMemoryModeHeap

Specify memory mode

After determining which memory mode is right for you, you must specify it when the SDK is initialized. Note that this must be done for audio. video, and share individually. To specify a raw data memory mode, provide one of these enums cases to the ZoomVideoSDKInitParams during SDK initialization.

ZoomVideoSDKInitParams *initParams = [[ZoomVideoSDKInitParams alloc] init];
// Set audio memory mode to heap.
initParams.audioRawdataMemoryMode = ZoomVideoSDKRawDataMemoryModeHeap;
// Set video memory mode to heap.
initParams.videoRawdataMemoryMode = ZoomVideoSDKRawDataMemoryModeHeap;
// Set share memory mode to heap.
initParams.shareRawdataMemoryMode = ZoomVideoSDKRawDataMemoryModeHeap;

ZoomVideoSDKInitParams *initParams = [[ZoomVideoSDKInitParams alloc] init];
// Set audio memory mode to heap.
initParams.audioRawdataMemoryMode = ZoomVideoSDKRawDataMemoryModeHeap;
// Set video memory mode to heap.
initParams.videoRawdataMemoryMode = ZoomVideoSDKRawDataMemoryModeHeap;
// Set share memory mode to heap.
initParams.shareRawdataMemoryMode = ZoomVideoSDKRawDataMemoryModeHeap;

Receive raw video data

Raw video data is offered in the following types:

  • YUV420: A data object commonly used by the renderer based on OpenGL ES.
  • CVPixelBuffer defined in NV12: A data object defined by Apple that can be used with Metal renderer.

To access and modify this data, you must:

  1. Implement an instance of the ZoomVideoSDKRawDataPipeDelegate.
  2. Use callback functions provided by the ZoomVideoSDKRawDataPipeDelegate to receive each frame of the raw video data.
  3. Pass the delegate into the video pipe of a specific user.

Implement ZoomVideoSDKRawDataPipeDelegate

// Used to receive video's NV12 data(CVPixelBufferRef).
- (void)onPixelBuffer:(CVPixelBufferRef)pixelBuffer rotation:(ZoomVideoSDKVideoRawDataRotation)rotation {
    // Access CVPixelBufferRef using pixelBuffer.
    // Get rotation of raw data video stream.
    switch (rotation) {
        case ZoomVideoSDKVideoRawDataRotation90:
            // 90 degrees.
            break;
        default:
            break;
    }
}
// Used to receive video's YUV420 data.
- (void)onRawDataFrameReceived:(ZoomVideoSDKVideoRawData *)rawData {
    // Access the raw data for each of the 3 components.
    char *yBuffer = [rawData yBuffer];
    char *uBuffer = [rawData uBuffer];
    char *vBuffer = [rawData vBuffer];
    // Get format of raw data.
    ZoomVideoSDKVideoRawDataFormat videoFormat = [rawData format];
    switch (videoFormat) {
        case ZoomVideoSDKVideoRawDataFormatI420:
            // Raw data is I420 format.
            break;
        case ZoomVideoSDKVideoRawDataFormatI420_Limit:
            // Raw data is I420_limit format.
            break;
    }
    // Get rotation of raw data video stream.
    ZoomVideoSDKVideoRawDataRotation rotation = [rawData rotation];
    switch (rotation) {
        case ZoomVideoSDKVideoRawDataRotation90:
            // 90 degrees.
            break;
        default:
            break;
    }
}
// Called when the raw data has been toggled.
- (void)onRawDataStatusChanged:(ZoomVideoSDKUserRawdataStatus)userRawdataStatus {
    switch (userRawdataStatus) {
        case ZoomVideoSDKUserRawdataOn:
            // User's raw data is now on.
            break;
        case ZoomVideoSDKUserRawdataOff:
            // User's raw data is now off.
            break;
    }
}
// Pass the delegate into the video pipe of a specific user (e.g. myself)
ZoomVideoSDKUser *user = [session getMySelf];
[user.getVideoPipe subscribeWithDelegate:delegate resolution:ZoomVideoSDKVideoResolution_720];

// Used to receive video's NV12 data(CVPixelBufferRef).
- (void)onPixelBuffer:(CVPixelBufferRef)pixelBuffer rotation:(ZoomVideoSDKVideoRawDataRotation)rotation {
    // Access CVPixelBufferRef using pixelBuffer.
    // Get rotation of raw data video stream.
    switch (rotation) {
        case ZoomVideoSDKVideoRawDataRotation90:
            // 90 degrees.
            break;
        default:
            break;
    }
}
// Used to receive video's YUV420 data.
- (void)onRawDataFrameReceived:(ZoomVideoSDKVideoRawData *)rawData {
    // Access the raw data for each of the 3 components.
    char *yBuffer = [rawData yBuffer];
    char *uBuffer = [rawData uBuffer];
    char *vBuffer = [rawData vBuffer];
    // Get format of raw data.
    ZoomVideoSDKVideoRawDataFormat videoFormat = [rawData format];
    switch (videoFormat) {
        case ZoomVideoSDKVideoRawDataFormatI420:
            // Raw data is I420 format.
            break;
        case ZoomVideoSDKVideoRawDataFormatI420_Limit:
            // Raw data is I420_limit format.
            break;
    }
    // Get rotation of raw data video stream.
    ZoomVideoSDKVideoRawDataRotation rotation = [rawData rotation];
    switch (rotation) {
        case ZoomVideoSDKVideoRawDataRotation90:
            // 90 degrees.
            break;
        default:
            break;
    }
}
// Called when the raw data has been toggled.
- (void)onRawDataStatusChanged:(ZoomVideoSDKUserRawdataStatus)userRawdataStatus {
    switch (userRawdataStatus) {
        case ZoomVideoSDKUserRawdataOn:
            // User's raw data is now on.
            break;
        case ZoomVideoSDKUserRawdataOff:
            // User's raw data is now off.
            break;
    }
}
// Pass the delegate into the video pipe of a specific user (e.g. myself)
ZoomVideoSDKUser *user = [session getMySelf];
[user.getVideoPipe subscribeWithDelegate:delegate resolution:ZoomVideoSDKVideoResolution_720];

Note: It is not recommended to implement a callback that does not belong to the renderer you are using. It is also not recommended to implement both onRawDataFrameReceived and onPixelBuffer. Implementing both callbacks will cause unnecessary and multiple raw data manipulations.

Remove backgrounds with an alpha channel

Session hosts who hold a raw streaming token can enable alpha channel mode, which requests that Zoom's Video SDK send both the original video data and an alpha mask demarking the edge of a user's body. The Video SDK can then use the alpha mask and raw data to remove the user's background and render session users natively in the host's virtual world.

Turn on alpha channel mode

Before turning on the alpha channel, call the following:

  • canEnableAlphaChannelMode to check whether the alpha channel mode is already turned on
  • isDeviceSupportAlphaChannelMode to check whether the device hardware capabilities are capable of supporting video alpha mode

To turn on, or off, the alpha channel, call enableAlphaChannelMode.

if let videoHelper = ZoomVideoSDK.shareInstance()?.getVideoHelper(), videoHelper.canEnableAlphaChannelMode() {
    videoHelper.isDeviceSupportAlphaChannelMode() // Determines whether the device hardware capabilities are capable of supporting video alpha mode.
    videoHelper.enableAlphaChannelMode(true) // true/false to enable/disable alpha channel
    videoHelper.isAlphaChannelModeEnabled()
}

ZoomVideoSDKVideoHelper *videoHelper = [[ZoomVideoSDK shareInstance] getVideoHelper];
if (videoHelper != nil) {
    if (videoHelper.canEnableAlphaChannelMode) {
        videoHelper.isDeviceSupportAlphaChannelMode; // Determines whether the device hardware capabilities are capable of supporting video alpha mode.
        [videoHelper enableAlphaChannelMode:TRUE]; // TRUE/FALSE to enable/disable alpha channel
        videoHelper.isAlphaChannelModeEnabled;
    }
}

Alpha channel callbacks

The onVideoAlphaChannelStatusChanged callback is triggered when alpha channel mode is turned on or off. After turning on the alpha channel, the raw video data will now have two more values alphaBuffer, for the alpha data buffer, and alphaBufferLen, for the alpha data buffer length, included in the same onRawDataFrameReceived callback.

Now that you've received the data, edit the raw data to suit your use case.

func onVideoAlphaChannelStatusChanged(_ isAlphaChannelOn: Bool) {
}
func onRawDataFrameReceived(_ rawData: ZoomVideoSDKVideoRawData?) {
    guard let rawData = rawData else { return }
    rawData.alphaBuffer // Only if alpha data exist, then will contains buffer address. Otherwise null.
    rawData.alphaBufferLen
}

- (void)onVideoAlphaChannelStatusChanged:(BOOL)isAlphaChannelOn {
}
- (void)onRawDataFrameReceived:(ZoomVideoSDKVideoRawData *)rawData {
    if (!rawData) {
        return;
    }
    rawData.alphaBuffer; // Only if alpha data exist, then will contains buffer address. Otherwise null.
    rawData.alphaBufferLen;
}

Receive raw audio data

Through your implementation of ZoomVideoSDKDelegate, you can access mixed (combined audio output from one or more users in a session), per-user, and shared raw audio data. Receive audio from hardware devices or virtual audio if it was sent through ZoomVideoSDKVirtualAudioMic.

Unlike raw video data, raw audio data will default to stack-based memory if you do not specify a memory mode.

  • The virtual speaker allows access to audio data received from other users in the session. This data represents what a user would hear played through the device's speakers.
  • The virtual mic allows audio data for the current user to be sent to the session programmatically instead of from the SDK capturing it through an audio input device.

Once you have implemented the memory mode callbacks, you must subscribe to the listener to receive audio data through an instance of ZoomVideoSDKAudioHelper.

- (void) onMixedAudioRawDataReceived:(ZoomVideoSDKAudioRawData *)rawData {
    // Access audio raw data for the whole session here.
}
- (void) onSharedAudioRawDataReceived:(ZoomVideoSDKAudioRawData *)rawData {
    // Access share audio raw data here.
}
- (void) onOneWayAudioRawDataReceived:(ZoomVideoSDKAudioRawData *)rawData user:(ZoomVideoSDKUser *)user {
    // Access per-user audio raw data here.
}

Receive raw audio for virtual speaker

Use the virtual speaker to process audio sent to the speaker and virtual microphone to process audio received from the microphone. If you receive or send audio directly, you won't be able to process it.

Follow these steps to receive virtual audio.

  1. Create an instance of ZoomVideoSDKVirtualAudioSpeaker.
  2. Pass that instance into ZoomVideoSDKSessionContext.
  3. Access raw data in each callback method.

See the following sample code for an example of how to receive raw audio data.

VirtualSpeakerExample.swift

import Foundation
class VirtualSpeakerExample: NSObject, ZoomVideoSDKVirtualAudioSpeaker {
    func onVirtualSpeakerMixedAudioReceived(_ rawData: ZoomVideoSDKAudioRawData!) {
        // Received raw audio from whole session that has been sent from virtual microphones
        // Play the raw audio here
    }
    func onVirtualSpeakerOneWayAudioReceived(_ rawData: ZoomVideoSDKAudioRawData!, user: ZoomVideoSDKUser!) {
        // Received raw audio from single user that has been sent from a virtual microphone
    }
    func onVirtualSpeakerSharedAudioReceived(_ rawData: ZoomVideoSDKAudioRawData!) {
        // Received raw audio from share that was manually sent
    }
}

VirtualSpeakerExample.swift

import Foundation
class VirtualSpeakerExample: NSObject, ZoomVideoSDKVirtualAudioSpeaker {
    func onVirtualSpeakerMixedAudioReceived(_ rawData: ZoomVideoSDKAudioRawData!) {
        // Received raw audio from whole session that has been sent from virtual microphones
        // Play the raw audio here
    }
    func onVirtualSpeakerOneWayAudioReceived(_ rawData: ZoomVideoSDKAudioRawData!, user: ZoomVideoSDKUser!) {
        // Received raw audio from single user that has been sent from a virtual microphone
    }
    func onVirtualSpeakerSharedAudioReceived(_ rawData: ZoomVideoSDKAudioRawData!) {
        // Received raw audio from share that was manually sent
    }
}

#import <Foundation/Foundation.h>
NS_ASSUME_NONNULL_BEGIN
@interface VirtualSpeakerExample : NSObject <ZoomVideoSDKVirtualAudioSpeaker>
@end
NS_ASSUME_NONNULL_END

Receive raw share data

Receive raw share video and audio data, for example, when someone is sharing their screen, similarly to how you receive raw video and raw audio data.

Receive raw share video data

Follow the same steps and code to receive raw video data, except get the share data instead of the video data, see the following code for an example.

// Previous code same as receive raw video
// Pass the ZoomVideoSDKRawDataPipeDelegate object into the share pipe of a specific user (e.g. myself)
ZoomVideoSDKUser *user = [session getMySelf];
[user.getSharePipe subscribeWithDelegate:delegate resolution:ZoomVideoSDKVideoResolution_720];

// Previous code same as receive raw video
// Pass the ZoomVideoSDKRawDataPipeDelegate object into the share pipe of a specific user (e.g. myself)
ZoomVideoSDKUser *user = [session getMySelf];
[user.getSharePipe subscribeWithDelegate:delegate resolution:ZoomVideoSDKVideoResolution_720];

Receive raw share audio data

Receive raw share audio data in the same way as you receive raw audio data.

Send raw video data

Send video data using sendVideoFrame within ZoomVideoSDKVideoSender. To obtain a ZoomVideoSDKVideoSender you must assign a ZoomVideoSDKVideoSource. The onInitialize within ZoomVideoSDKVideoSource will provide a ZoomVideoSDKVideoSender.

- (void)onInitialize:(ZoomVideoSDKVideoSender *)rawDataSender supportCapabilityArray:(NSArray *)supportCapabilityArray suggestCapability:(ZoomVideoSDKVideoCapability *)suggestCapabilityItem {
    // Store video rawdata sender.
    self.videoRawdataSender = rawDataSender;
}
// Call sendVideoFrame to send a frame buffer of raw data.
[self.videoRawdataSender sendVideoFrame:frameBuffer width:width height:height dataLength:dataLength rotation:rotation];

- (void)onInitialize:(ZoomVideoSDKVideoSender *)rawDataSender supportCapabilityArray:(NSArray *)supportCapabilityArray suggestCapability:(ZoomVideoSDKVideoCapability *)suggestCapabilityItem {
    // Store video rawdata sender.
    self.videoRawdataSender = rawDataSender;
}
// Call sendVideoFrame to send a frame buffer of raw data.
[self.videoRawdataSender sendVideoFrame:frameBuffer width:width height:height dataLength:dataLength rotation:rotation];

In the onInitialize:(ZoomVideoSDKVideoSender *)rawDataSender supportCapabilityArray:(NSArray *)supportCapabilityArray suggestCapability:(ZoomVideoSDKVideoCapability *)suggestCapabilityItem callback:

  • The supportCapabilityArray parameter is the supported capability list. It combines the supported resolution of the session and the device to create a list of supported capabilities.
  • The suggestCapabilityItem parameter is the suggested capability. It combines the maximum capability of the session itself and the device to get the real maximum as the suggested capability.

Pre-process raw video data

Pre-process raw video data using onPreProcessRawData within ZoomVideoSDKVideoSourcePreProcessor.

func onPreProcessRawData(_ rawData: ZoomVideoSDKPreProcessRawData!) {
    // Perform preprocess actions here.
}

- (void)onPreProcessRawData:(ZoomVideoSDKPreProcessRawData *)rawData {
    // Perform preprocess actions here.
}

Send raw audio data

Follow these steps to send a user's raw audio or audio that you have processed from a user's device.

  1. Create an instance of ZoomVideoSDKVirtualAudioMic.
  2. Pass that instance into ZoomVideoSDKSessionContext.
  3. Obtain ZoomVideoSDKAudioSender from the onMicInitialize callback.
  4. Send raw audio data with the send method.

VirtualMicExample.h

#import <Foundation/Foundation.h>
NS_ASSUME_NONNULL_BEGIN
@interface VirtualMicExample : NSObject <ZoomVideoSDKVirtualAudioMic>
@property (nonatomic, strong) ZoomVideoSDKAudioSender *audioSender;
@end
NS_ASSUME_NONNULL_END

VirtualMicExample.swift

class VirtualMicExample: NSObject, ZoomVideoSDKVirtualAudioMic {
    var audioSender: ZoomVideoSDKAudioSender?
    func onMicInitialize(_ rawDataSender: ZoomVideoSDKAudioSender) {
        // Virtual Microphone has initialized, store rawDataSender to send raw audio data into session later
        audioSender = rawDataSender
    }
    func onMicStartSend() {
        guard let audioSender = audioSender else { return }
        // Virtual Microphone can begin sending raw audio into session
        // Provide your audio data info here
        guard let yourAudioBuffer = UnsafeMutablePointer<Int8>(mutating: ("buffer here" as NSString).utf8String) else { return }
        let yourAudioDataLength = UInt()
        let yourAudioSampleRate = UInt()
        let sendRawAudioReturnStatus = audioSender.send(yourAudioBuffer, dataLength: yourAudioDataLength, sampleRate: yourAudioSampleRate)
        switch sendRawAudioReturnStatus {
        case .Errors_Success:
            print("Call to send raw audio succeeded")
        default:
            print("Call to send raw audio produced an error: \(sendRawAudioReturnStatus)")
        }
    }
    func onMicStopSend() {
        // Virtual Microphone has stopped sending raw audio
    }
    func onMicUninitialized() {
        // Virtual Microphone has been destroyed
        audioSender = nil
    }
}
// Create a VirtualMicExample object and assign it to the ZoomVideoSDKSessionContext object virtualAudioMicDelegate you have created before joining a session.
let virtualAudioMic = VirtualMicExample()
sessionContext.virtualAudioMicDelegate = virtualAudioMic

VirtualMicExample.swift

class VirtualMicExample: NSObject, ZoomVideoSDKVirtualAudioMic {
    var audioSender: ZoomVideoSDKAudioSender?
    func onMicInitialize(_ rawDataSender: ZoomVideoSDKAudioSender) {
        // Virtual Microphone has initialized, store rawDataSender to send raw audio data into session later
        audioSender = rawDataSender
    }
    func onMicStartSend() {
        guard let audioSender = audioSender else { return }
        // Virtual Microphone can begin sending raw audio into session
        // Provide your audio data info here
        guard let yourAudioBuffer = UnsafeMutablePointer<Int8>(mutating: ("buffer here" as NSString).utf8String) else { return }
        let yourAudioDataLength = UInt()
        let yourAudioSampleRate = UInt()
        let sendRawAudioReturnStatus = audioSender.send(yourAudioBuffer, dataLength: yourAudioDataLength, sampleRate: yourAudioSampleRate)
        switch sendRawAudioReturnStatus {
        case .Errors_Success:
            print("Call to send raw audio succeeded")
        default:
            print("Call to send raw audio produced an error: \(sendRawAudioReturnStatus)")
        }
    }
    func onMicStopSend() {
        // Virtual Microphone has stopped sending raw audio
    }
    - (void)onMicStopSend;
    {
        // Virtual Microphone has stopped sending raw audio
    }
    - (void)onMicUninitialized;
    {
        // Virtual Microphone has been destroyed
        self.audioSender = nil;
    }
    @end
    // Create a VirtualMicExample object and assign it to the ZoomVideoSDKSessionContext object virtualAudioMicDelegate you have created before joining a session.
    VirtualMicExample *virtualAudioMic = [VirtualMicExample new];
    sessionContext.virtualAudioMicDelegate = virtualAudioMic;
}

Send raw share data

Follow these steps to send raw screen share data.

  1. Have your class conform to the IZoomVideoSDKShareSource protocol.
  2. Use the sendShareFrame method in onShareSendStarted to send raw screen share data.

RawShareDataExample.h

#import <Foundation/Foundation.h>
@interface RawShareDataExample : NSObject <ZoomVideoSDKShareSource>
@property (nonatomic, strong) ZoomVideoSDKShareSender *shareSender;
@end

@implementation RawShareDataExample
- (void)onShareSendStarted:(ZoomVideoSDKShareSender *)rawDataSender
{
    self.shareSender = rawDataSender;
    // sendShareFrame sends one frame data, therefore you will most likely need a thread for the full frame.
    [self.shareSender sendShareFrame:(char *) width:(int) height:(int) frameLength:(int) format:(ZoomVideoSDKFrameDataFormat)];
}
- (void)onShareSendStopped {
    // Stop sending raw share data
    self.shareSender = nil;
    // If you have a thread for onShareSendStarted, you will most likely need to cancel it and set it to nil.
}

RawShareDataExample.m

#import "RawShareDataExample.h"
@implementation RawShareDataExample
- (void)onShareSendStarted:(ZoomVideoSDKShareSender *)rawDataSender
{
    self.shareSender = rawDataSender;
    // sendShareFrame sends one frame data, therefore you will most likely need a thread for the full frame.
    [self.shareSender sendShareFrame:(char *) width:(int) height:(int) frameLength:(int) format:(ZoomVideoSDKFrameDataFormat)];
}
- (void)onShareSendStopped {
    // Stop sending raw share data
    self.shareSender = nil;
    // If you have a thread for onShareSendStarted, you will most likely need to cancel it and set it to nil.
}