Receive raw data

You receive a remote user's raw video and share frames by subscribing to their pipe with a ZMVideoSDKRawDataPipeDelegate, and raw audio through your ZMVideoSDKDelegate callbacks or a virtual speaker.

Before receiving raw data, set the raw data memory mode.

Receive raw video data

Raw video data is encoded in the YUV420p format. YUV420 is a data object commonly used by the renderer based on OpenGL ES.

Follow these steps to access and modify the raw video data.

  1. Implement an instance of the ZMVideoSDKRawDataPipeDelegate.
  2. Use callback functions provided by the ZMVideoSDKRawDataPipeDelegate to receive each frame of the raw video data.
  3. Pass the delegate into the video pipe of a specific user.
// Used to receive video's YUV420 data.
func onRawDataFrameReceived(_ data: ZMVideoSDKYUVRawDataI420!) {
    // Access the raw data for each of the 3 components.
    let yBuffer = data.yBuffer
    let uBuffer = data.uBuffer
    let vBuffer = data.vBuffer
    // Get format of raw data.
    let isVideoFormatLimitedI420 = data.isLimitedI420
    // Get rotation of raw data video stream.
    let rotation = data.rotation
}
// Called when the raw data has been toggled.
func onRawDataStatusChanged(_ status: ZMVideoSDKRawDataStatus) {
    switch userRawdataStatus {
        case ._On:
        // User's raw data is now on.
        break
        case ._Off:
        // User's raw data is now off.
        break
    }
}
// Pass the delegate into the video pipe of a specific user (e.g. myself).
let user = session.getMySelf()
user.getVideoPipe()?.subscribe(with: delegate, resolution: ._720)

// Used to receive video's YUV420 data.
- (void)onRawDataFrameReceived:(ZMVideoSDKYUVRawDataI420*)data {
    // Access the raw data for each of the 3 components.
    char *yBuffer = [data yBuffer];
    char *uBuffer = [data uBuffer];
    char *vBuffer = [data vBuffer];
    // Get format of raw data.
    bool isVideoFormatLimitedI420 = [data isLimitedI420];
    // Get rotation of raw data video stream.
    unsigned int rotation = [data rotation];
}
// Called when the raw data has been toggled.
- (void)onRawDataStatusChanged:(ZMVideoSDKRawDataStatus)status {
    switch (status) {
        case ZMVideoSDKRawData_On:
            // User's raw data is now on.
            break;
        case ZMVideoSDKRawData_Off:
            // User's raw data is now off.
            break;
    }
}
// Pass the delegate into the video pipe of a specific user (e.g. myself).
ZMVideoSDKUser *user = [session getMySelf];
[user.getVideoPipe subscribe:ZMVideoSDKResolution_720P listener:delegate];

Receive raw audio data

Through your implementation of ZMVideoSDKDelegate, you can access mixed (combined audio output from one or more users in a session), per-user, and shared raw audio data. Receive audio from hardware devices or virtual audio if it was sent through ZMVideoSDKVirtualAudioMic.

Unlike raw video data, raw audio data will default to stack-based memory if you do not specify a memory mode.

  • The virtual speaker allows access to audio data received from other users in the session. This data represents what a user would hear played through the device's speakers.
  • The virtual mic allows audio data for the current user to be sent to the session programmatically instead of from the SDK capturing it through an audio input device.

Once you have implemented the memory mode callbacks, subscribe to the listener to receive audio data through an instance of ZMVideoSDKAudioHelper.

func onMixedAudioRawDataReceived(_ data: ZMVideoSDKAudioRawData!) {
    // Access audio raw data for the whole session here.
}
func onSharedAudioRawDataReceived(_ data: ZMVideoSDKAudioRawData!) {
    // Access share audio raw data here.
}
func onOneWayAudioRawDataReceived(_ data: ZMVideoSDKAudioRawData!, user: ZMVideoSDKUser!) {
    // Access per-user audio raw data here.
}

- (void) onMixedAudioRawDataReceived:(ZMVideoSDKAudioRawData *)data {
    // Access audio raw data for the whole session here.
}
- (void) onSharedAudioRawDataReceived:(ZMVideoSDKAudioRawData *)data {
    // Access share audio raw data here.
}
- (void) onOneWayAudioRawDataReceived:(ZMVideoSDKAudioRawData *)data user:(ZMVideoSDKUser *)user {
    // Access per-user audio raw data here.
}

Receive raw audio for virtual speaker

Use virtual speaker to process audio sent to the speaker and virtual microphone to process audio received from the microphone. If you receive or send audio directly, you won't be able to process it.

Follow these steps to receive raw audio.

  1. Create an instance of ZMVideoSDKVirtualAudioSpeaker.
  2. Pass that instance into ZMVideoSDKSessionContext.
  3. Access raw data in each callback method.

To access raw audio data, listen for the following callbacks in your listener.

func onVirtualSpeakerMixedAudioReceived(_ rawdata: ZMVideoSDKAudioRawData) {
    // Access audio raw data for the whole session here.
}
func onVirtualSpeakerOneWayAudioReceived(_ rawdata: ZMVideoSDKAudioRawData, user: ZMVideoSDKUser) {
    // Access per-user audio raw data here.
}
func onVirtualSpeakerSharedAudioReceived(_ rawdata: ZMVideoSDKAudioRawData) {
    // Access share audio raw data here.
}

- (void) onVirtualSpeakerMixedAudioReceived:(ZMVideoSDKAudioRawData *)data {
    // Access audio raw data for the whole session here.
}
- (void) onVirtualSpeakerSharedAudioReceived:(ZMVideoSDKAudioRawData *)data {
    // Access share audio raw data here.
}
- (void) onVirtualSpeakerOneWayAudioReceived:(ZMVideoSDKAudioRawData *)data user:(ZMVideoSDKUser *)user {
    // Access per-user audio raw data here.
}

Receive raw share data

Receive raw share video and audio data, for example, when someone is sharing their screen, similarly to how you receive raw video and raw audio data.

Receive raw share video data

Follow the same steps and code to receive raw video data, except get the share data instead of the video data. See the following code for an example.

// Previous code same as receive raw video.
// Pass the ZMVideoSDKRawDataPipeDelegate object into the share pipe of a specific user (e.g. myself).
let user = session.getMySelf()
user.getSharePipe()?.subscribe(with: delegate, resolution: ._720)

// Previous code same as receive raw video.
// Pass the ZMVideoSDKRawDataPipeDelegate object into the share pipe of a specific user (e.g. myself).
ZMVideoSDKUser *user = [session getMySelf];
[user.getSharePipe subscribeWithDelegate:delegate resolution:ZMVideoSDKVideoResolution_720];

Receive raw share audio data

Receive raw share audio data in the same way as you receive raw audio data.