Filtering camera preview with Metal
In this post, we will use all knowledge acquired from the metal filtering lesson to filter camera preview on iPhone.
I have created a starter project. It will display a camera preview using the front camera on iPhone or iPad. Implementation is straightforward. I'm using a basic, well-known image capture setup with a preview layer. It is working, but we want to have Comic Filter applied to the camera preview.
From the previous post, I'll reuse the whole MetalView with some small changes. We'll need to dispatch rendering to the main thread and create new init
because Metal View will be created from code, not from Storyboard. Add these changes to MetalView
:
var image: CIImage? {
didSet {
DispatchQueue.main.async {
self.drawCIImge()
}
}
}
init() {
super.init(frame: .zero, device: MTLCreateSystemDefaultDevice())
self.isOpaque = false
self.framebufferOnly = false
self.enableSetNeedsDisplay = true
self.context = CIContext(mtlDevice: device!)
self.queue = device!.makeCommandQueue()
}
Create a new file, FilteringCameraController
. I'll write a new camera controller that uses the MetalView
and it will add comic effect to preview output. To do that, the controller will have to capture the sample buffer in the delegate and display it manually on preview view. This will be a little bit more complicated, but with code from the Metal tutorial, the rendering part is already done. We just have to prepare filtered CIImage
to be displayed by the metal view.
I'll post the whole class and explain the code point by point.
//
// FilteringCameraController.swift
//
import Foundation
import AVFoundation
import CoreGraphics
import CoreImage
import UIKit
class FilteringCameraController: NSObject {
private var previewView = MetalView()
@objc dynamic private let captureSession = AVCaptureSession()
private let captureSessionQueue = DispatchQueue(label: "FilteringCameraController_capture_session_queue",
attributes: [])
private var videoInput: AVCaptureDeviceInput?
private var videoOutput: AVCaptureVideoDataOutput?
private var setupComplete = false
private var captureVideoOrientation = AVCaptureVideoOrientation.portrait
// 1
private lazy var filter: CIFilter = {
let filterInternal = CIFilter(name: "CIComicEffect")!
return filterInternal
}()
var flashMode = AVCaptureDevice.FlashMode.auto
// 2
func prepareCamera(with previewView: UIView) {
if setupComplete || AVCaptureDevice.authorizationStatus(for: .video) == .denied {
return
}
previewView.addSubview(self.previewView)
self.previewView.frame = previewView.frame
previewView.bringSubviewToFront(self.previewView)
setupInput(for: .front)
setupComplete = true
}
// 3
private func setupInput(for cameraPosition: AVCaptureDevice.Position) {
captureSessionQueue.async {
self.prepareInput(for: cameraPosition)
self.setupOutputs()
if self.captureSession.canSetSessionPreset(.photo) {
self.captureSession.sessionPreset = .photo
}
}
}
// 4
private func prepareInput(for cameraPosition: AVCaptureDevice.Position) {
guard let videoDevice = captureDevice(with: AVMediaType.video.rawValue, position: cameraPosition) else {
return
}
let videoDeviceInput: AVCaptureDeviceInput!
do {
videoDeviceInput = try AVCaptureDeviceInput(device: videoDevice)
} catch {
fatalError(error.localizedDescription)
}
if self.captureSession.canAddInput(videoDeviceInput) {
self.captureSession.addInput(videoDeviceInput)
self.videoInput = videoDeviceInput
}
}
// 5
private func setupOutputs() {
let videoDataOutput = AVCaptureVideoDataOutput()
videoDataOutput.setSampleBufferDelegate(self, queue: self.captureSessionQueue)
if self.captureSession.canAddOutput(videoDataOutput) {
self.captureSession.addOutput(videoDataOutput)
self.videoOutput = videoDataOutput
}
}
// 6
func startCamera() {
if !setupComplete {
return
}
if captureSession.isRunning {
return
}
captureSessionQueue.async { [unowned self] in
self.captureSession.startRunning()
}
}
// 7
private func captureDevice(with mediaType: String, position: AVCaptureDevice.Position?) -> AVCaptureDevice? {
let session = AVCaptureDevice.DiscoverySession(deviceTypes: [.builtInWideAngleCamera], mediaType: AVMediaType.video, position: .unspecified)
let cameras = session.devices
var captureDevice = cameras.first
if let position = position {
for device in cameras where device.position == position {
captureDevice = device
break
}
}
if position == .back {
try? captureDevice?.lockForConfiguration()
captureDevice?.focusMode = .continuousAutoFocus
captureDevice?.unlockForConfiguration()
}
return captureDevice
}
enum CameraControllerError: Swift.Error {
case captureSessionAlreadyRunning
case captureSessionIsMissing
case inputsAreInvalid
case invalidOperation
case noCamerasAvailable
case unknown
}
}
// 8
extension FilteringCameraController: AVCaptureVideoDataOutputSampleBufferDelegate, AVCaptureAudioDataOutputSampleBufferDelegate {
public func captureOutput(_ captureOutput: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
guard let imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else {
return
}
let sourceImage = CIImage(cvPixelBuffer: imageBuffer as CVPixelBuffer)
filter.setValue(sourceImage, forKey: kCIInputImageKey)
let filteredImage = filter.outputImage!
// 9
let rotation = -CGFloat.pi
let translationX: CGFloat = -1
let rotated = filteredImage
.transformed(by: CGAffineTransform(rotationAngle: rotation / 2))
.transformed(by: CGAffineTransform(scaleX: translationX, y: 1))
let transformed = rotated.transformed(by: .init(translationX: -rotated.extent.origin.x,
y: -rotated.extent.origin.y))
// 10
previewView.image = transformed
}
}
- CIFilter
The lazy created comic filter will be used with a camera buffer to create an output image. - Entry method
Before using the camera, we have to callprepareCamera(with previewView: UIView)
. It will handle creating inputs, outputs, and delegate setup. - Our code will use a separate queue for handling all camera-related actions. We need to be in sync with the camera to perform actions related to the setup.
- Our capture session needs input. In our example, I'll use the front camera as input.
- Unlike provided implementation, our new camera controller will use video data output. The output will call the delegate every time the sample buffer is ready.
- After successful preparation, the camera can be started and will begin to produce an output buffer.
- Helper method for creating capture device
It uses a discovery session to retrieve available capture devices (front and back cameras). In our code, only the front camera will be used. You can experiment and use a back camera if you like. - Delegate
In this example, we just care about video output by implementingcaptureOutput(_ captureOutput: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection)
. Core Image provides convince method for creatingCIImage
from pixel buffer. After that, it is used as an input for the comic filter. - Scaling and rotating
By default, camera output will be rotated 90º and will not be mirrored. To make it looks natural on-screen, the code needs to rotate it-π
, and then transform it by scaling to make it mirrored. We also need to translate the image to make it in0,0
position. This will ensure proper rendering. Add a breakpoint and use Xcode to see how the image changes in every step, Xcode will render aCIImage
preview for you. - Rendering
This line of code will render our camera preview. I told you, it will work out of the box!
The last thing to do is to change the View controller to accommodate the new camera controller.
Change camera controller:
let cameraController = FilteringCameraController()
Remove code from view did load, it is not needed anymore.
override func viewDidLoad() {
super.viewDidLoad()
}
Our setup will take place in view will appear:
override func viewWillAppear(_ animated: Bool) {
super.viewWillAppear(animated)
cameraController.prepareCamera(with: self.view)
cameraController.startCamera()
}
Build and run the application. How awesome is the result? You can experiment with filters and produce your own, unique effect! Let your creativity go crazy:)
If you want to get the final result, it is on GitHub.