How ARKit 2 Handles Occlusion in iOS 12

In iOS 12, Apple introduced ARKit 2 with improved tracking and environment understanding, but it did not yet possess hardware-based LiDAR sensors for automated, per-pixel depth occlusion. Instead, ARKit 2 achieved occlusion by combining computer vision tracking—such as horizontal and vertical plane detection and 3D object recognition—with custom rendering techniques using invisible geometric proxies and depth-buffer manipulation. This allowed developers to hide virtual assets realistically behind recognized real-world boundaries.

The Role of the Depth Buffer and Occlusion Materials

At the rendering level, ARKit 2 relies on the graphics pipeline's depth buffer (Z-buffer) to determine visibility. To hide a digital object behind a physical one, developers use an "occlusion material."

In frameworks like SceneKit or Metal, an occlusion material renders geometry invisible to the human eye while still writing its depth coordinates to the Z-buffer. When a virtual object is placed in the scene, the rendering engine checks whether its depth value is behind the invisible geometry's depth value. If it is further away, those pixels are discarded, creating the illusion that the real-world object is covering the virtual model.

Plane Detection as Physical Barriers

ARKit 2 expanded surface tracking to detect both horizontal surfaces (such as floors and tabletops) and vertical surfaces (such as walls and doors).

When ARKit detects a plane, it provides the position, orientation, and extent of the surface. Developers use this data to dynamically generate flat polygonal meshes aligned with the real surface. By applying an occlusion material to these planes, any virtual object that drops below a table or moves behind a wall is automatically culled from the camera's view.

3D Object Detection and Geometric Proxies

A major feature introduced in ARKit 2 was the ability to scan, recognize, and track known three-dimensional physical objects using ARObjectScanningConfiguration and ARReferenceObject.

Once the device detects a pre-scanned physical object in the physical environment, ARKit reports its exact 3D position and orientation. Developers can attach a pre-modeled 3D "proxy mesh" matching the dimensions of the real object directly over it. Because this proxy mesh uses occlusion shaders, any virtual item placed or animated behind that physical object is correctly obscured.

Facial Occlusion with TrueDepth

For front-facing camera experiences on supported devices (such as iPhone X), ARKit 2 utilized the TrueDepth sensor to generate a live, highly accurate 3D mesh of the user's face. By treating this face mesh as an occluder, virtual objects like glasses frames or digital hats could wrap naturally around the head, disappearing behind the ears, nose, or cheeks instead of appearing awkwardly superimposed.