What’s New in RealityKit 2
Customize shaders, meshes, materials, animations, audio, and more in the latest release of Apple’s Augmented Reality framework
This article originally appeared on Codrs at Tumblr.
The first session I watched today at WWDC 2021 was Dive into RealityKit 2, an overview of the updates to Apple’s newest augmented reality framework. At WWDC Community Week, GoodNotes highlighted sketchnoting in a tutorial session, so I thought I’d give this format a try and provide a quick visualization of the new updates. I’ve been writing RealityKit tutorials since its initial release at WWDC 2019 with my series RealityKit Snippets, and its been my 3D graphics framework of choice since then for AR apps.
Custom Shaders & Meshes
RealityKit 2 brings the framework to the next level with custom shaders and meshes, features that let developers customize materials and graphics and generate 3D shapes/content on the fly. This includes the steaming pizzas and confetti that Apple highlighted at the Keynote on Monday. While RealityKit remains a high-level and approachable framework to use, the new customization options provide control over content that used to require SceneKit or Metal.
Faces
With the new mesh features, you can access face meshes from the TrueDepth camera and modify the materials and texture. For example, the session showed off a face painting app with glitter. This feature will be great for AR photo filters that need to place content that fits to the face’s shape.
Audio
Spatial Audio also received as mention, with a new AudioBufferResource that lets you create spatial-aware sounds dynamically instead of loading from a static file.
Materials
I’ve been waiting for transparent materials in AR for over three years, and I’m so excited that it’s finally here! Image and video textures can now include alpha channels, so 2D content can have irregular borders. Going along with the theme of customization, RealityKit 2 provides full control over physically based materials from the USD format.
Entity-Component System
The session also highlighted updates to the Entity-Component System, the architecture that RealityKit apps follow to organize apps. The session introduced Systems, a way to control 3D models, listen for updates, and coordinate events. EntityQuery is a new way to search a scene for entities with specified components, a great task for a System to manage.
The session encouraged developers to include all state and properties for objects in Components instead of subclassing Entity. Entity should be used a container of components with an identifier, so all information about the appearance, animation, behaviors, and other state should belong in components.
Animation
It is now possible to blend multiple animations together, adjust the speed, and work with animation timelines. Again, it will be easier to tailor animations beyond what a USDZ already contains.
Character Controller
Leveraging a 3D reconstruction of the world from the LiDAR on iPad Pro and iPhone 12 Pro, Scene Understanding makes AR experiences even more realistic by allowing virtual content to collide with and be occluded by the real world. In my NSSpain 2020 presentation, I built an AR racing game with Scene Understanding.
New this year, a Character Controller makes it easier to work with user-controlled objects. Simply define a CapsuleShape that takes the shape and size of the 3D model to detect collisions with the world around you. Then you can call the move(to:) method to have the character navigate around real obstacles on its way to the final destination. Or, if you’d prefer a more direct path, teleport(to:) will move the object without paying attention to objects in the way.
That is all the major announcements included in the Dive into RealityKit 2 session! I will be writing more about the new sessions later this week, which will go into more detail about these new features.
