027ba3860839bb567c3e41e20af21bfc
如何利用 AVFoundation 设计一个通用稳定的音视频框架?

前言

承接上篇的《AV Foundation开发秘籍——实践掌握iOS & OS X应用的视听处理技术 阅读指南》
今天这篇给大家讲解下如何利用AVFoundation设计一套通用稳定的音视频框架。

核心

AVCaptureSession开启捕获任务,配置AVCaptureDeviceInput定制捕获任务的输入源(多种摄像头),通过AVFoundation内各种Data output输出数据(元数据、视频帧、音频帧),AVAssetWriter开启写任务,将音视频数据归档为媒体文件。

实现功能

视频流预览、录像归档、捕获相片、切换摄像头、人脸检测、帧率配置、相机详细配置

框架源码

https://github.com/caixindong/XDCaptureService

具体设计

核心模块 XDCaptureService & XDVideoWritter

XDCaptureService是对外API的总入口,也是框架的核心类,主要做音视频输入输出配置工作和调度工作。
XDVideoWritter是音视频写模块,主要提供写数据和归档数据的基础操作,不对外暴露。
对外API设计:

@class XDCaptureService;


@protocol XDCaptureServiceDelegate <NSObject>

@optional
//service生命周期
- (void)captureServiceDidStartService:(XDCaptureService *)service;

- (void)captureService:(XDCaptureService *)service serviceDidFailWithError:(NSError *)error;

- (void)captureServiceDidStopService:(XDCaptureService *)service;

- (void)captureService:(XDCaptureService *)service getPreviewLayer:(AVCaptureVideoPreviewLayer *)previewLayer;

- (void)captureService:(XDCaptureService *)service outputSampleBuffer:(CMSampleBufferRef)sampleBuffer;

//录像相关
- (void)captureServiceRecorderDidStart:(XDCaptureService *)service ;

- (void)captureService:(XDCaptureService *)service recorderDidFailWithError:(NSError *)error;

- (void)captureServiceRecorderDidStop:(XDCaptureService *)service;

//照片捕获
- (void)captureService:(XDCaptureService *)service capturePhoto:(UIImage *)photo;

//人脸检测
- (void)captureService:(XDCaptureService *)service outputFaceDetectData:(NSArray <AVMetadataFaceObject*>*) faces;

//景深数据
- (void)captureService:(XDCaptureService *)service captureTrueDepth:(AVDepthData *)depthData API_AVAILABLE(ios(11.0));

@end

@protocol XDCaptureServicePreViewSource <NSObject>

- (AVCaptureVideoPreviewLayer *)preViewLayerSource;

@end

@interface XDCaptureService : NSObject

//是否录制音频,默认是NO
@property (nonatomic, assign) BOOL shouldRecordAudio;

//iOS原生人脸检测,默认是NO
@property (nonatomic, assign) BOOL openNativeFaceDetect;

//摄像头的方向,默认是AVCaptureDevicePositionFront(前置)
@property (nonatomic, assign) AVCaptureDevicePosition devicePosition;

//判断是否支持景深模式,当前只支持7p、8p、X的后置摄像头及X的前后摄像头,系统要求是iOS 11以上
@property (nonatomic, assign, readonly) BOOL depthSupported;

//是否开启景深模式,默认是NO
@property (nonatomic, assign) BOOL openDepth;

//只有以下指定的sessionPreset才有depth数据:AVCaptureSessionPresetPhoto、AVCaptureSessionPreset1280x720、AVCaptureSessionPreset640x480
@property (nonatomic, assign) AVCaptureSessionPreset sessionPreset;

//帧率,默认是30
@property (nonatomic, assign) int frameRate;

//录像的临时存储地址,建议每次录完视频做下重定向
@property (nonatomic, strong, readonly) NSURL *recordURL;

//如果设置preViewSource则内部不生成AVCaptureVideoPreviewLayer
@property (nonatomic, assign) id<XDCaptureServicePreViewSource> preViewSource;

@property (nonatomic, assign) id<XDCaptureServiceDelegate> delegate;

@property (nonatomic, assign, readonly) BOOL isRunning;


//视频编码设置(影响录制的视频的编码和大小)
@property (nonatomic, strong) NSDictionary *videoSetting;

///相机专业设置,除非特定需求,一般不设置
//感光度(iOS8以上)
@property (nonatomic, assign, readonly) CGFloat deviceISO;
@property (nonatomic, assign, readonly) CGFloat deviceMinISO;
@property (nonatomic, assign, readonly) CGFloat deviceMaxISO;

//镜头光圈大小
@property (nonatomic, assign, readonly) CGFloat deviceAperture;

//曝光
@property (nonatomic, assign, readonly) BOOL supportsTapToExpose;
@property (nonatomic, assign) AVCaptureExposureMode exposureMode;
@property (nonatomic, assign) CGPoint exposurePoint;
@property (nonatomic, assign, readonly) CMTime deviceExposureDuration;

//聚焦
@property (nonatomic, assign, readonly) BOOL supportsTapToFocus;
@property (nonatomic, assign) AVCaptureFocusMode focusMode;
@property (nonatomic, assign) CGPoint focusPoint;

//白平衡
@property (nonatomic, assign) AVCaptureWhiteBalanceMode whiteBalanceMode;

//手电筒
@property (nonatomic, assign, readonly) BOOL hasTorch;
@property (nonatomic, assign) AVCaptureTorchMode torchMode;

//闪光灯
@property (nonatomic, assign, readonly) BOOL hasFlash;
@property (nonatomic, assign) AVCaptureFlashMode flashMode;

//相机权限判断
+ (BOOL)videoGranted;

//麦克风权限判断
+ (BOOL)audioGranted;

//切换摄像机
- (void)switchCamera;

//启动
- (void)startRunning;

//关闭
- (void)stopRunning;

//开始录像
- (void)startRecording;

//取消录像
- (void)cancleRecording;

//停止录像
- (void)stopRecording;

//拍照
- (void)capturePhoto;

@end

CDG队列分流

因为在主线程启动音视频捕获及音视频读写会阻塞主线程,所以我们需要将这些任务派发到子线程中执行。我们选择GCD队列帮我们做这个视频。我们框架总共配置3个队列,分别是sessionQueue、writtingQueue、outputQueue,这些队列都是串行队列,因为音视频相关操作都是有顺序(时序)要求,保证当前队列只有一个操作的执行(配置、写数据、读数据)。sessionQueue主要负责音视频任务启动的调度,writtingQueue主要负责写数据的调度,保证数据帧能够准确归档到文件,outputQueue主要负责数据帧的输出。

@property (nonatomic, strong) dispatch_queue_t sessionQueue;
@property (nonatomic, strong) dispatch_queue_t writtingQueue;
@property (nonatomic, strong) dispatch_queue_t outputQueue;

 _sessionQueue = dispatch_queue_create("com.caixindong.captureservice.session", DISPATCH_QUEUE_SERIAL);
_writtingQueue = dispatch_queue_create("com.caixindong.captureservice.writting", DISPATCH_QUEUE_SERIAL);
_outputQueue = dispatch_queue_create("com.caixindong.captureservice.output", DISPATCH_QUEUE_SERIAL);

音视频捕获

初始化捕获任务

sessionPreset指定了输出的视频帧的像素,例如640*480

@property (nonatomic, strong) AVCaptureSession *captureSession;
 _captureSession = [[AVCaptureSession alloc] init];
_captureSession.sessionPreset = _sessionPreset;

配置捕获的输入

获取输入源设备,通过_cameraWithPosition可以获取摄像头的抽象表示,因为红外摄像头、双摄像头只能从较新API中获取,所以方法里已经做了兼容处理。并用输入源设备配置捕获输入AVCaptureDeviceInput

@property (nonatomic, strong) AVCaptureDeviceInput *videoInput;

- (BOOL)_setupVideoInputOutput:(NSError **) error {
    self.currentDevice = [self _cameraWithPosition:_devicePosition];

    self.videoInput = [[AVCaptureDeviceInput alloc] initWithDevice:_currentDevice error:error];
    if (_videoInput) {
        if ([_captureSession canAddInput:_videoInput]) {
            [_captureSession addInput:_videoInput];
        } else {
            *error = [NSError errorWithDomain:@"com.caixindong.captureservice.video" code:-2200 userInfo:@{NSLocalizedDescriptionKey:@"add video input fail"}];
            return NO;
        }
    } else {
        *error = [NSError errorWithDomain:@"com.caixindong.captureservice.video" code:-2201 userInfo:@{NSLocalizedDescriptionKey:@"video input is nil"}];
        return NO;
    }

    //稳定帧率
    CMTime frameDuration = CMTimeMake(1, _frameRate);
    if ([_currentDevice lockForConfiguration:error]) {
        _currentDevice.activeVideoMaxFrameDuration = frameDuration;
        _currentDevice.activeVideoMinFrameDuration = frameDuration;
        [_currentDevice unlockForConfiguration];
    } else {
        *error = [NSError errorWithDomain:@"com.caixindong.captureservice.video" code:-2203 userInfo:@{NSLocalizedDescriptionKey:@"device lock fail(input)"}];

        return NO;
    }

……Other code
}

- (AVCaptureDevice *)_cameraWithPosition:(AVCaptureDevicePosition)position {
    if (@available(iOS 10.0, *)) {
        //AVCaptureDeviceTypeBuiltInWideAngleCamera默认广角摄像头,AVCaptureDeviceTypeBuiltInTelephotoCamera长焦摄像头,AVCaptureDeviceTypeBuiltInDualCamera后置双摄像头,AVCaptureDeviceTypeBuiltInTrueDepthCamera红外前置摄像头
        NSMutableArray *mulArr = [NSMutableArray arrayWithObjects:AVCaptureDeviceTypeBuiltInWideAngleCamera,AVCaptureDeviceTypeBuiltInTelephotoCamera,nil];
        if (@available(iOS 10.2, *)) {
            [mulArr addObject:AVCaptureDeviceTypeBuiltInDualCamera];
        }
        if (@available(iOS 11.1, *)) {
            [mulArr addObject:AVCaptureDeviceTypeBuiltInTrueDepthCamera];
        }
        AVCaptureDeviceDiscoverySession *discoverySession = [AVCaptureDeviceDiscoverySession discoverySessionWithDeviceTypes:[mulArr copy] mediaType:AVMediaTypeVideo position:position];
        return discoverySession.devices.firstObject;
    } else {
        NSArray *videoDevices = [AVCaptureDevice devicesWithMediaType:AVMediaTypeVideo];
        for (AVCaptureDevice *device in videoDevices) {
            if (device.position == position) {
                return device;
            }
        }
    }
    return nil;
}

配置捕获输出

根据不同的功能需求,我们可以往捕获任务里添加不同的输出,例如捕获基础的视频帧数据,我们添加AVCaptureVideoDataOutput,捕获音频数据,我们添加AVCaptureAudioDataOutput,捕获人脸数据,我们添加AVCaptureMetadataOutput。因为音频的输出和视频输出的设置方式大同小异,所以这里只列出视频输出的关键代码,这里有几个关键的设计:
1、因为相机传感器问题,输出的视频流的方向会有90度偏转,所以我们需要通过获取与输出连接的videoConnection进行偏转配置;
2、视频帧(或者音频帧)都是以CMSampleBufferRef格式输出,视频帧可能经过多个业务处理,例如写文件或者抛到上层业务处理,所以处理数据前都对视频帧数据进行retatin操作,保证各个业务线处理的视频帧是独立的,具体可以看_processVideoData;
3、为了及时清理临时变量(对视频帧处理的各种操作可能需要较多内存空间),所以将外抛的帧处理用autorelease pool包裹起来,防止出现内存高峰;

@property (nonatomic, strong) AVCaptureVideoDataOutput *videoOutput;

- (BOOL)_setupVideoInputOutput:(NSError **) error {
……Other code

self.videoOutput = [[AVCaptureVideoDataOutput alloc] init];
    _videoOutput.videoSettings = @{(id)kCVPixelBufferPixelFormatTypeKey: @(kCVPixelFormatType_32BGRA)};
    //对迟到的帧做丢帧处理
    _videoOutput.alwaysDiscardsLateVideoFrames = YES;

    dispatch_queue_t videoQueue = dispatch_queue_create("com.caixindong.captureservice.video", DISPATCH_QUEUE_SERIAL);
    //设置数据输出的delegate
    [_videoOutput setSampleBufferDelegate:self queue:videoQueue];

    if ([_captureSession canAddOutput:_videoOutput]) {
        [_captureSession addOutput:_videoOutput];
    } else {
        *error = [NSError errorWithDomain:@"com.caixindong.captureservice.video" code:-2204 userInfo:@{NSLocalizedDescriptionKey:@"device lock fail(output)"}];
        return NO;
    }

    self.videoConnection = [self.videoOutput connectionWithMediaType:AVMediaTypeVideo];
    //录制视频会有90度偏转,是因为相机传感器问题,所以在这里设置输出的视频流的方向
    if (_videoConnection.isVideoOrientationSupported) {
        _videoConnection.videoOrientation = AVCaptureVideoOrientationPortrait;
    }
    return YES;
}

#pragma mark - AVCaptureVideoDataOutputSampleBufferDelegate && AVCaptureAudioDataOutputSampleBufferDelegate
- (void)captureOutput:(AVCaptureOutput *)output didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection {
    //可以捕获到不同的线程
    if (connection == _videoConnection) {
        @synchronized(self) {
            [self _processVideoData:sampleBuffer];
        };
    } else if (connection == _audioConnection) {
        @synchronized(self) {
            [self _processAudioData:sampleBuffer];
        };
    }
}

#pragma mark - process Data
- (void)_processVideoData:(CMSampleBufferRef)sampleBuffer {
    //CFRetain的目的是为了每条业务线(写视频、抛帧)的sampleBuffer都是独立的
    if (_videoWriter && _videoWriter.isWriting) {
        CFRetain(sampleBuffer);
        dispatch_async(_writtingQueue, ^{
            [_videoWriter appendSampleBuffer:sampleBuffer];
            CFRelease(sampleBuffer);
        });
    }

    CFRetain(sampleBuffer);
    //及时清理临时变量,防止出现内存高峰
    dispatch_async(_outputQueue, ^{
        @autoreleasepool{
            if (self.delegate && [self.delegate respondsToSelector:@selector(captureService:outputSampleBuffer:)]) {
                [self.delegate captureService:self outputSampleBuffer:sampleBuffer];
            }
        }
        CFRelease(sampleBuffer);
    });
}

配置图片数据输出

配置图片数据输出的目的是为了实现相片捕获功能,通过setOutputSettings,我们可以配置我们输出的图片格式。

@property (nonatomic, strong) AVCaptureStillImageOutput *imageOutput;

- (BOOL)_setupImageOutput:(NSError **)error {
    self.imageOutput = [[AVCaptureStillImageOutput alloc] init];
    NSDictionary *outputSetting = @{AVVideoCodecKey: AVVideoCodecJPEG};
    [_imageOutput setOutputSettings:outputSetting];
    if ([_captureSession canAddOutput:_imageOutput]) {
        [_captureSession addOutput:_imageOutput];
        return YES;
    } else {
        *error = [NSError errorWithDomain:@"com.caixindong.captureservice.image" code:-2205 userInfo:@{NSLocalizedDescriptionKey:@"device lock fail(output)"}];
        return NO;
    }
}

//拍照功能实现
- (void)capturePhoto {
    AVCaptureConnection *connection = [_imageOutput connectionWithMediaType:AVMediaTypeVideo];
    if (connection.isVideoOrientationSupported) {
        connection.videoOrientation = AVCaptureVideoOrientationPortrait;
    }

    __weak typeof(self) weakSelf = self;
    [_imageOutput captureStillImageAsynchronouslyFromConnection:connection completionHandler:^(CMSampleBufferRef  _Nullable imageDataSampleBuffer, NSError * _Nullable error) {
        __strong typeof(weakSelf) strongSelf = weakSelf;
        if (imageDataSampleBuffer != NULL) {
            NSData *imageData = [AVCaptureStillImageOutput jpegStillImageNSDataRepresentation:imageDataSampleBuffer];
            UIImage *image = [UIImage imageWithData:imageData];
            if (strongSelf.delegate && [strongSelf.delegate respondsToSelector:@selector(captureService:capturePhoto:)]) {
                [strongSelf.delegate captureService:strongSelf capturePhoto:image];
            }
        }
    }];
}

配置人脸数据输出

关键是配置人脸元数据输出AVCaptureMetadataOutput,并指定metadataObjectTypes为AVMetadataObjectTypeFace,捕获的人脸数据包含当前视频帧中所有的人脸,可以从数据中提取人脸的范围、位置、偏转角,但这个有个注意点,就是原始的人脸数据的坐标是相机坐标系,我们需要转化为屏幕坐标,这样才方便我们的业务处理,具体可以看人脸数据输出那一块。

top Created with Sketch.