View with 2 videos is not getting recorded

[nihtin9mk]

I am using ASScreenRecorder in my app. My view controller has 2 views both playing videos - just like video chat in skype application.

When I try to record my screen - screen is recorded with only one video view the other one is missing in the video why it is happening ??

[alskipp]

Hi @nihtin9mk,
There are certain types of view/layer that can't be automatically recorded - AVCaptureVideoPreviewLayer is one such case. To record these views you need to manually write the pixel data into the CGContext using the writeBackgroundFrameInContext:(CGContextRef*)contextRef method.

As you described your use case as a 'video chat' application, I assume this is probably the problem you are encountering. Is one of the video views a live preview?

Al

[nihtin9mk]

@alskipp
Hi, yes exactly this is the issue in my app. The view is a live preview.

Please give me a solution for this.
I am so much thankful if you can provide some sample codes.

In my view controller I wrote the below codes

  -(void)viewDidLoad
   {
    [super viewDidLoad];
     [ASScreenRecorder sharedInstance].delegate = self;

    }


//myVideoView is the imageview with video preiview

  -(void)writeBackgroundFrameInContext:(CGContextRef *)contextRef{

   CGSize imgSize = [myVideoView.image size];
   UIGraphicsBeginImageContext(imgSize);
   contextRef = UIGraphicsGetCurrentContext();

    }

I believe this is not the proper way - please guide me.

[nihtin9mk]

@alskipp Plz help me to do this

[alskipp]

Hi I'll do my best to point you in the right direction either later today or tomorrow. As you can probably appreciate this open source library doesn't pay the bills and I'm not independently wealthy, consequently I'm currently working for 'the man'.

As a quick pointer - you need to get direct access to the pixel data in your live video input. If memory serves correctly you'll need to implement a method in AVCaptureVideoDataOutputSampleBufferDelegate. Probably the easiest thing to do is to create a CGImage iVar in your controller that you continuously update in -captureOutput:didOutputSampleBuffer:FromConnection.

You then implement the delgate method like y

[alskipp]

Whoops. Typing this on phone and accidentally tapped close and comment.

Anyway. You need to implement the delegate method, but you then draw the CGImage (which you created in captureOutput:didOutputSampleBuffer… into the context.

[nihtin9mk]

Hi @alskipp - Thank you for your help in this, and of course all your open source works are really helpful and great for the developers like me.

I am not so much familiar with AVFoundation and captureOutput:didOutputSampleBuffer things. Hope you can give a better picture on how to implement the delegate method.
Please take your time.

And once again a lot of thanks for your selfless work.

[alskipp]

Hi, I'm on my way back home now. I'll try and post a few code examples this evening or tomorrow. If you get chance take a look at the documentation for AVCaptureVideoDataOutputSampleBufferDelegate as I think your view controller will need to implement this delegate to receive the live video data. You'll then use this pixel data to write into the video context.

Al

[alskipp]

I'll give a code example of how to turn the pixel data from captureOutput:didOutputSampleBuffer:FromConnection into an image you can use.

[alskipp]

OK. Here we go:
First your view controller will need to implement AVCaptureVideoDataOutputSampleBufferDelegate - declared something like:
@interface CameraViewController : UIViewController <AVCaptureVideoDataOutputSampleBufferDelegate>

As you're already previewing the video on screen I assume there's already an AVCaptureSession set up which is then used to init a AVCaptureVideoPreviewLayer.

In viewDidLoad you'll want to register to receive a notification for when your captureSession starts - something like:

[[NSNotificationCenter defaultCenter] addObserver:self
                                         selector:@selector(captureBegan)
                                             name:AVCaptureSessionDidStartRunningNotification
                                           object:self.captureSession];

Then you'll need to implement the method named in the selector:

- (void)captureBegan
{
    [ASScreenRecorder sharedInstance].delegate = self;
}

Don't forget to remove the delegate when the captureSession ends [ASScreenRecorder sharedInstance].delegate = nil.

The next comment will show the basics of AVCaptureVideoDataOutputSampleBufferDelegate it won't be the full implementation you need just yet. But let's just try and get an image created from the sampleBuffer and then we can continue from there…

[alskipp]

This doesn't do anything too useful yet - but let's see if the CGImageRef is successfully created, if so most of the hard work has been achieved, there are just a few more pieces to put in place.

- (void)captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection
{
    CGImageRef image = [self createCGImageFromSampleBuffer:sampleBuffer];
    // check if image creation is successful
    CGImageRelease(image);
}

Here is how to get a CGImage from the sampleBuffer:

- (CGImageRef)createCGImageFromSampleBuffer:(CMSampleBufferRef) sampleBuffer
{
    CVImageBufferRef imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
    CVPixelBufferLockBaseAddress(imageBuffer,0);

    void *baseAddress = CVPixelBufferGetBaseAddress(imageBuffer);
    size_t bytesPerRow = CVPixelBufferGetBytesPerRow(imageBuffer);
    size_t width = CVPixelBufferGetWidth(imageBuffer);
    size_t height = CVPixelBufferGetHeight(imageBuffer);

    CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceRGB();

    CGContextRef imageContext = CGBitmapContextCreate(baseAddress, width, height, 8, bytesPerRow, colorSpace, kCGBitmapByteOrder32Little | kCGImageAlphaPremultipliedFirst);

    CGImageRef newImage = CGBitmapContextCreateImage(imageContext);

    CGContextRelease(imageContext);
    CGColorSpaceRelease(colorSpace);
    CVPixelBufferUnlockBaseAddress(imageBuffer,0);

    return newImage;
}

Let me know if all this works so far. Then I'll continue with the final bits that write the captured image into the video buffer.

(for now don't call [ASScreenRecorder sharedInstance].delegate = self; as we've not implemented the delegate method yet, so the compiler will complain. We'll get to that bit next).
Good luck.

[alskipp]

Just realised I missed out a vital bit. Sorry! We need to declare ourselves as delegate to AVCaptureVideoDataOutput. Otherwise we won't receive any calls from captureOutput:didOutputSampleBuffer:FromConnection. I'll post some code examples in a moment.

[alskipp]

First we need a dispatch queue to receive calls to captureOutput:didOutputSampleBuffer:FromConnection - we don't want to block the main thread with video processing.

@property (strong, nonatomic) dispatch_queue_t videoQueue;

We then initialize the queue and use it when we declare ourselves as delegate to the captureSession output.

Here's some code that shows how the capture session is setup:

- (void)setupCamera
{
    AVCaptureDevice *captureDevice = [AVCaptureDevice defaultDeviceWithMediaType:AVMediaTypeVideo];

    if (![captureDevice hasMediaType:AVMediaTypeVideo]) {
        return;
    }

    _videoQueue = dispatch_queue_create("CameraViewController.videoQueue", DISPATCH_QUEUE_SERIAL);

    AVCaptureDeviceInput *input = [AVCaptureDeviceInput deviceInputWithDevice:captureDevice error:nil];

    self.captureSession = [[AVCaptureSession alloc] init];

    AVCaptureVideoDataOutput *output = [[AVCaptureVideoDataOutput alloc] init];

    output.videoSettings = @{ (NSString*)kCVPixelBufferPixelFormatTypeKey : @(kCVPixelFormatType_32BGRA) };
    [output setSampleBufferDelegate:self queue:_videoQueue];

    [self.captureSession addOutput:output];
    [self.captureSession addInput:input];
    [self.captureSession setSessionPreset:AVCaptureSessionPreset640x480];

    _previewLayer = [[AVCaptureVideoPreviewLayer alloc] initWithSession:self.captureSession];

    [self.cameraView setVideoPreviewLayer:_previewLayer];
}

The vital bit that you need to receive the sampleBuffers is:

[output setSampleBufferDelegate:self queue:_videoQueue];

Once that is declared you should start receiving calls to captureOutput:didOutputSampleBuffer:FromConnection