Video Materials in RealityKit

Add videos to planes, boxes, spheres, and more in augmented reality

Among the RealityKit updates from WWDC 2020 are video materials. Like other RealityKit materials, VideoMaterial can be applied to any mesh, accepting an AVPlayer to handle playback. Here is a guide on how to add video to any RealityKit AR experience.

Add a video to your project

1. Add a video file to your iOS target’s main folder
2. Go to the iOS target’s settings, select Build Phases, and add your video under Copy Bundle Resources.

Load the video in an AVPlayer

RealityKit uses an AVPlayer for video materials, so make sure to add import AVKit at the top of your ViewController.swift file.

To get the path of the video file, change the forResource and ofType parameters in the code to match your file. For example, if your file is myFileName.mp4 , the resource is myFileName , and the type is mp4 . With the path, we create a URL and then an AVPlayerItem to manage timing. Finally, create an AVPlayer .

let videoURL = URL(fileURLWithPath: path)
let playerItem = AVPlayerItem(url: videoURL)
let videoPlayer = AVPlayer(playerItem: playerItem)

Afterwards, creating a VideoMaterial from the AVPlayer is very simple.

let videoMaterial = VideoMaterial(avPlayer: videoPlayer)

Create the material, mesh, and anchor.

Your video needs a mesh to attach to, and RealityKit provides a number of primitive shapes to choose from. This example uses a plane, but you can create some interesting effects with other shapes. I attached the plane mesh to a vertical plane anchor so that it looks like a TV on the wall. Afterwards, don’t forget to add the anchor to the scene!

let videoPlane = ModelEntity(mesh: .generatePlane(width: 1.6, depth: 0.9), materials: [videoMaterial])
let anchor = AnchorEntity(plane: .vertical)
anchor.addChild(videoPlane)
arView.scene.anchors.append(anchor)

Play the video

Video playback is managed by the AVPlayer rather than the VideoMaterial. To start the video, simply call the play method on the player.

videoPlayer.play()

Looping playback

This code will only play the video once, but in many cases you will want to video to keep playing. To accomplish this, you can listen to notifications when the AVPlayerItem has reached the end of the video.

First, you’ll need to store the AVPlayer as a property on the view controller to access it from outside the method you created it in. After creating a videoPlayer property, set it to the video player you created earlier.

class ViewController: UIViewController {
    var videoPlayer: AVPlayer! // Add this line
    
    override func viewDidLoad() {
        let videoPlayer = AVPlayer(playerItem: playerItem)
        self.videoPlayer = videoPlayer // Add this line
    }
}

Notification Center allows you to listen to an event, like an AVPlayerItem finishing a video. After creating the video player, add a notification observer. The selector refers to a method that we will create next to handle the looping.

NotificationCenter.default.addObserver(
    self, 
    selector: #selector(loopVideo), 
    name: NSNotification.Name.AVPlayerItemDidPlayToEndTime, 
    object: playerItem
)

The loopVideo method simply takes the AVPlayerItem from the notification, moves the video back the the start, and hits play again.

@objc func loopVideo(notification: Notification) {
    guard let playerItem = notification.object as? AVPlayerItem else  { return }
    playerItem.seek(to: CMTime.zero, completionHandler: nil)        
    videoPlayer.play()
}

Full code example

For more RealityKit or WWDC 2020 coverage, see my articles below. Thank you!

ARKit @ WWDC 2020

RealityKit Snippets