Customize shaders, meshes, materials, animations, audio, and more in the latest release of Apple’s Augmented Reality framework

This article originally appeared on Codrs at Tumblr.

Custom Shaders & Meshes

RealityKit 2 brings the framework to the…


Recaps, code samples, and resources for new Core ML and Create ML announcements at WWDC 2021!

My memoji sits on the WWDC 2021 banner with an iMessage bubble reading ML@WWDC
My memoji sits on the WWDC 2021 banner with an iMessage bubble reading ML@WWDC

I’ll be updating this page throughout the week with resources to get started with the new machine learning related announcements at WWDC, so keep checking back and follow me on Twitter for updates.

In the meantime, check out last year’s updates here.


Recaps, code samples, and resources for new ARKit and RealityKit announcements at WWDC 2021!

My memoji sits on top of the WWDC 2021 banner with an iMessage bubble reading AR@WWDC.
My memoji sits on top of the WWDC 2021 banner with an iMessage bubble reading AR@WWDC.

I’ll be updating this page throughout the week with resources to get started with the new AR related announcements at WWDC, so keep checking back and follow me on Twitter for updates.

Dive into RealityKit 2 Session

I sketchnoted the Dive into RealityKit 2 Session with all the important points you need to know to get started with the new features. Read all the details here!


Bring augmented reality into real world locations with geo-tracking from WWDC 2020

How geo-tracking works

As part of rebuilding the data backend for Apple Maps, Apple collected camera and 3D LiDAR data from city streets around the world. When using location anchors, ARKit downloads the virtual map surrounding your device from the cloud and matches it with the device’s camera feed. Combined with GPS, ARKit can quickly and precisely determine your…


Add videos to planes, boxes, spheres, and more in augmented reality

Among the RealityKit updates from WWDC 2020 are video materials. Like other RealityKit materials, VideoMaterial can be applied to any mesh, accepting an AVPlayer to handle playback. Here is a guide on how to add video to any RealityKit AR experience.

Add a video to your project

  1. Add a video file to your iOS target’s main folder
  2. Go to the iOS target’s settings, select Build Phases, and add your video under Copy Bundle Resources.


Recaps, code samples, and resources for new Machine Learning announcements at WWDC 2020!

WWDC 2020 here, and Apple has upgraded its Machine Learning tools and platforms. I will be updating this article with summaries of the announcements, code samples, and tutorials with the latest technologies during the conference week!

New in Core ML

  1. Deploy models to apps with CloudKit
  2. Model encryption

New in Create ML

  1. Train video models
  2. Train style transfer models on both image and video

Core ML Tools

Core ML Tools are a Python library for creating machine learning models in the Core ML format, and it received a huge update this week. The tools make it possible to convert models from TensorFlow (both 1.x and 2.x) …


Recaps, code samples, and resources for new ARKit announcements at WWDC 2020!

WWDC 2020 is here with ARKit 4 and RealityKit 2. Here is a running list of all new features with tutorials and code samples added throughout the week!

New at the Keynote

ARKit 4 Features

— Depth API for LiDAR on iPad Pro

— Apple Maps integration for real world location anchoring

— Face Tracking without TrueDepth camera

Tutorials


Since COVID-19 forced the closing of nearly all Apple Stores worldwide in March, Apple fans and customers have been shopping and admiring the latest products from the comfort and safety of their homes.

Starting with the Mac Pro announced at WWDC 2019, Apple has provided an AR Quick Look experience for new products to visualize them in your own space. However, I wanted to bring the entire store experience to the home and immerse users in a room full of shiny new iPhone, iPad, and Mac devices.


Enhance your AR apps with spatial audio in RealityKit/ARKit

This is part of my RealityKit Snippets collection of short code examples for getting things done in RealityKit.

Audio in RealityKit

To make your augmented reality experiences even more realistic, RealityKit has tools to incorporate sound. The sound capabilities include:

  • Loading and playing audio files
  • Adjusting sound to user orientation
  • Attaching sound to physical locations (The sound level changes as you move closer to/further away from an entity)

Add audio to an entity

Options

inputMode changes how the sound reacts to the real world.

  • .nonSpatial plays the audio normally regardless of position and orientation.
  • .ambient adjusts the left/right audio balance to follow which way the user is facing.


Ray casting is the new hit testing in ARKit 3

This is part of my RealityKit Snippets collection of short code examples for getting things done in RealityKit.

About Ray Casting & Applications

  • Convert from 2D screen space to 3D surfaces in world space
  • Place content on a surface directly in front of the camera or where the user taps the screen

Simple Ray Cast Example

Make a ray cast from the center of the screen.

Tracked Ray Cast Example

ARKit will keep refining the ray cast as it learns more about detected surfaces.

Modifications

  1. Ray cast origin (from: CGPoint)
    Change this parameter to any point on the screen. …

Ethan Saadia

18 year old developer working on innovative, creative technologies with both hardware and software.  WWDC Scholar 2019 & Swift Student Challenge Winner 2020.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store