# Receive raw data You receive a remote user's raw video and share frames by subscribing to their pipe with a `ZoomVideoSDKRawDataPipeDelegate`, and raw audio through your `ZoomVideoSDKDelegate` callbacks or a virtual speaker. ## Receive raw video data Raw video data is offered in the following types: - `YUV420`: A data object commonly used by the renderer based on OpenGL ES. - `CVPixelBuffer defined in NV12`: A data object defined by Apple that can be used with a Metal renderer. To access and modify this data, you must: 1. Implement an instance of the `ZoomVideoSDKRawDataPipeDelegate`. 2. Use the callback functions provided by the `ZoomVideoSDKRawDataPipeDelegate` to receive each frame of the raw video data. 3. Pass the delegate into the video pipe of a specific user. ### Implement ZoomVideoSDKRawDataPipeDelegate ```swift // Used to receive video's NV12 data (CVPixelBufferRef). func onPixelBuffer(_ pixelBuffer: CVPixelBuffer!, rotation: ZoomVideoSDKVideoRawDataRotation) { // Access CVPixelBufferRef using pixelBuffer. // Get rotation of raw data video stream. switch rotation { case ZoomVideoSDKVideoRawDataRotation90: // 90 degrees. break default: break } } // Used to receive video's YUV420 data. func onRawDataFrameReceived(_ rawData: ZoomVideoSDKVideoRawData!) { // Access the raw data for each of the 3 components — see "Read a video frame" below. } // Called when the sender stops or starts sending raw data. func onRawDataStatusChanged(_ userRawdataStatus: ZoomVideoSDKUserRawdataStatus) { switch userRawdataStatus { case ZoomVideoSDKUserRawdataOn: // User's raw data is now on. break case ZoomVideoSDKUserRawdataOff: // User's raw data is now off. break default: break } } ``` ```objectivec // Used to receive video's NV12 data (CVPixelBufferRef). - (void)onPixelBuffer:(CVPixelBufferRef)pixelBuffer rotation:(ZoomVideoSDKVideoRawDataRotation)rotation { // Access CVPixelBufferRef using pixelBuffer. // Get rotation of raw data video stream. switch (rotation) { case ZoomVideoSDKVideoRawDataRotation90: // 90 degrees. break; default: break; } } // Used to receive video's YUV420 data. - (void)onRawDataFrameReceived:(ZoomVideoSDKVideoRawData *)rawData { // Access the raw data for each of the 3 components — see "Read a video frame" below. } // Called when the sender stops or starts sending raw data. - (void)onRawDataStatusChanged:(ZoomVideoSDKUserRawdataStatus)userRawdataStatus { switch (userRawdataStatus) { case ZoomVideoSDKUserRawdataOn: // User's raw data is now on. break; case ZoomVideoSDKUserRawdataOff: // User's raw data is now off. break; } } ``` > **Note** > > It is not recommended to implement a callback that does not belong to the renderer you are using. It is also not recommended to implement both `onRawDataFrameReceived` and `onPixelBuffer`. Implementing both callbacks will cause unnecessary and multiple raw data manipulations. To read the per-pixel alpha mask alongside the YUV frame, see [Raw video with alpha channel](/docs/video-sdk/ios/raw-data/raw-video-alpha-channel/). ### Subscribe to a user's video pipe To start receiving frames, pass your delegate into the video pipe of a specific user and check the return value. ```swift guard let user = session.getMySelf(), let pipe = user.getVideoPipe() else { return } let result = pipe.subscribe(with: delegate, resolution: ZoomVideoSDKVideoResolution_720) if result != .Errors_Success { // Subscription was rejected. } ``` ```objectivec ZoomVideoSDKUser *user = [session getMySelf]; ZoomVideoSDKError result = [user.getVideoPipe subscribeWithDelegate:delegate resolution:ZoomVideoSDKVideoResolution_720]; if (result != Errors_Success) { // Subscription was rejected. } ``` `subscribeWithDelegate:resolution:` returns a `ZoomVideoSDKError`. `Errors_Success` means frames will start arriving on `onRawDataFrameReceived` (or `onPixelBuffer`); any other value is an error code and no frames are delivered. A concrete resolution is required — `Auto` is not accepted for raw-data subscriptions. ### Read a video frame Inside `onRawDataFrameReceived`, read the planes and metadata from the `ZoomVideoSDKVideoRawData`. ```swift let size = rawData.size // CGSize — frame width and height let yBuffer = rawData.yBuffer // char * for the Y plane let uBuffer = rawData.uBuffer // char * for the U plane let vBuffer = rawData.vBuffer // char * for the V plane let rotation = rawData.rotation // ZoomVideoSDKVideoRawDataRotationNone / ...90 / ...180 / ...270 let format = rawData.format // ZoomVideoSDKFrameDataFormat_I420 or ..._I420_Limit let timeStamp = rawData.timeStamp // NSDate ``` ```objectivec CGSize size = [rawData size]; // frame width and height char *yBuffer = [rawData yBuffer]; // Y plane char *uBuffer = [rawData uBuffer]; // U plane char *vBuffer = [rawData vBuffer]; // V plane ZoomVideoSDKVideoRawDataRotation rotation = [rawData rotation]; ZoomVideoSDKFrameDataFormat format = [rawData format]; NSDate *timeStamp = [rawData timeStamp]; switch (format) { case ZoomVideoSDKFrameDataFormat_I420: // Raw data is I420 format. break; case ZoomVideoSDKFrameDataFormat_I420_Limit: // Raw data is I420 (limited range) format. break; } ``` The frame's dimensions come from `size` (a `CGSize`); there are no separate `streamWidth` / `streamHeight` accessors on iOS. | Accessor | Type | Description | | ----------- | ---------------------------------- | ------------------------------------------------------------------------------- | | `size` | `CGSize` | Frame width and height in pixels. | | `yBuffer` | `char *` | Y (luma) plane. | | `uBuffer` | `char *` | U (chroma) plane. | | `vBuffer` | `char *` | V (chroma) plane. | | `rotation` | `ZoomVideoSDKVideoRawDataRotation` | `…RotationNone` (0), `…Rotation90`, `…Rotation180`, `…Rotation270` (clockwise). | | `format` | `ZoomVideoSDKFrameDataFormat` | `ZoomVideoSDKFrameDataFormat_I420` or `ZoomVideoSDKFrameDataFormat_I420_Limit`. | | `timeStamp` | `NSDate *` | Capture time of the frame. | ### Retain a frame past the callback Raw data buffers are valid only inside `onRawDataFrameReceived` by default. Once the callback returns, the buffer may be recycled. To hold onto a frame — for example, to hand it off to a background thread for processing or encoding — increase its reference count with `addRef`, then release it with `releaseRef` when you are done. ```swift func onRawDataFrameReceived(_ rawData: ZoomVideoSDKVideoRawData!) { guard rawData.canAddRef() else { return } rawData.addRef() backgroundQueue.async { defer { rawData.releaseRef() } self.processFrame(rawData) } } ``` ```objectivec - (void)onRawDataFrameReceived:(ZoomVideoSDKVideoRawData *)rawData { if (![rawData canAddRef]) { return; } [rawData addRef]; dispatch_async(self.backgroundQueue, ^{ [self processFrame:rawData]; [rawData releaseRef]; }); } ``` Check `canAddRef` before calling `addRef`, and always pair `addRef` with `releaseRef`. Orphaned refs leak frame buffers. ### Stop receiving frames To stop receiving frames, call `unSubscribeWithDelegate:` on the same pipe with the delegate you subscribed. Always unsubscribe before tearing down the owning view controller, otherwise the SDK keeps invoking your delegate against a deallocated context. ```swift user.getVideoPipe()?.unSubscribe(with: delegate) ``` ```objectivec [user.getVideoPipe unSubscribeWithDelegate:delegate]; ``` ## Receive raw audio data Through your implementation of `ZoomVideoSDKDelegate`, you can access mixed (combined audio output from one or more users in a session), per-user, and shared raw audio data. Receive audio from hardware devices or virtual audio if it was sent through `ZoomVideoSDKVirtualAudioMic`. > Unlike raw video data, raw audio data will default to stack-based memory if you do not specify a memory mode. - The **virtual speaker** allows access to audio data received from other users in the session. This data represents what a user would hear played through the device's speakers. - The **virtual mic** allows audio data for the current user to be sent to the session programmatically instead of from the SDK capturing it through an audio input device. Once you have implemented the memory mode callbacks, you must subscribe to the listener to receive audio data through an instance of `ZoomVideoSDKAudioHelper`. ```swift func onMixedAudioRawDataReceived(_ rawData: ZoomVideoSDKAudioRawData!) { // Access audio raw data for the whole session here. } func onSharedAudioRawDataReceived(_ rawData: ZoomVideoSDKAudioRawData!) { // Access share audio raw data here. } func onOneWayAudioRawDataReceived(_ rawData: ZoomVideoSDKAudioRawData!, user: ZoomVideoSDKUser!) { // Access per-user audio raw data here. } ``` ```objectivec - (void)onMixedAudioRawDataReceived:(ZoomVideoSDKAudioRawData *)rawData { // Access audio raw data for the whole session here. } - (void)onSharedAudioRawDataReceived:(ZoomVideoSDKAudioRawData *)rawData { // Access share audio raw data here. } - (void)onOneWayAudioRawDataReceived:(ZoomVideoSDKAudioRawData *)rawData user:(ZoomVideoSDKUser *)user { // Access per-user audio raw data here. } ``` Inside each callback, the `ZoomVideoSDKAudioRawData` exposes the PCM buffer through `buffer` (with `bufferLen`), along with `sampleRate` and `channelNum`. ### Receive raw audio for virtual speaker Use the virtual speaker to process audio sent to the speaker, and the virtual microphone to process audio received from the microphone. If you receive or send audio directly, you won't be able to process it. Follow these steps to receive virtual audio. 1. Create an instance of `ZoomVideoSDKVirtualAudioSpeaker`. 2. Pass that instance into `ZoomVideoSDKSessionContext`. 3. Access raw data in each callback method. **`VirtualSpeakerExample.swift`** ```swift import Foundation class VirtualSpeakerExample: NSObject, ZoomVideoSDKVirtualAudioSpeaker { func onVirtualSpeakerMixedAudioReceived(_ rawData: ZoomVideoSDKAudioRawData!) { // Received raw audio from the whole session that was sent from virtual microphones. // Play the raw audio here. } func onVirtualSpeakerOneWayAudioReceived(_ rawData: ZoomVideoSDKAudioRawData!, user: ZoomVideoSDKUser!) { // Received raw audio from a single user that was sent from a virtual microphone. } func onVirtualSpeakerSharedAudioReceived(_ rawData: ZoomVideoSDKAudioRawData!) { // Received raw audio from share that was manually sent. } } ``` **`VirtualSpeakerExample.h`** ```objectivec #import NS_ASSUME_NONNULL_BEGIN @interface VirtualSpeakerExample : NSObject @end NS_ASSUME_NONNULL_END ``` **`VirtualSpeakerExample.m`** ```objectivec #import "VirtualSpeakerExample.h" @implementation VirtualSpeakerExample - (void)onVirtualSpeakerMixedAudioReceived:(ZoomVideoSDKAudioRawData *)rawData { // Received raw audio from the whole session that was sent from virtual microphones. // Play the raw audio here. } - (void)onVirtualSpeakerOneWayAudioReceived:(ZoomVideoSDKAudioRawData *)rawData user:(ZoomVideoSDKUser *)user { // Received raw audio from a single user that was sent from a virtual microphone. } - (void)onVirtualSpeakerSharedAudioReceived:(ZoomVideoSDKAudioRawData *)rawData { // Received raw audio from share that was manually sent. } @end ``` ## Receive raw share data Receive raw share video and audio data, for example when someone is sharing their screen, similarly to how you receive raw video and raw audio data. ### Receive raw share video data Follow the same steps and code to receive raw video data, except get the share pipe instead of the video pipe, as shown in the following code. ```swift // Previous code same as receive raw video. // Pass the delegate into the share pipe of a specific user (for example, myself). guard let user = session.getMySelf(), let pipe = user.getSharePipe() else { return } let result = pipe.subscribe(with: delegate, resolution: ZoomVideoSDKVideoResolution_720) if result != .Errors_Success { // Subscription was rejected. } ``` ```objectivec // Previous code same as receive raw video. // Pass the ZoomVideoSDKRawDataPipeDelegate object into the share pipe of a specific user (for example, myself). ZoomVideoSDKUser *user = [session getMySelf]; ZoomVideoSDKError result = [user.getSharePipe subscribeWithDelegate:delegate resolution:ZoomVideoSDKVideoResolution_720]; if (result != Errors_Success) { // Subscription was rejected. } ``` The same `onRawDataFrameReceived` callback fires for share frames; the same accessors, reference-counting, and unsubscribe rules apply. ### Receive raw share audio data Receive raw share audio data in the same way as you receive raw audio data, through the `onSharedAudioRawDataReceived` callback on your `ZoomVideoSDKDelegate`, or `onVirtualSpeakerSharedAudioReceived` on a `ZoomVideoSDKVirtualAudioSpeaker`.