How iOS 12 Renders Realistic AR Shadows

With the release of iOS 12 and ARKit 2, Apple significantly elevated the visual fidelity of augmented reality by enabling virtual objects to cast convincing, lifelike shadows onto physical surfaces. This article breaks down the underlying technologies that make this possible, including environmental light estimation, invisible shadow-catcher planes, contact shadow modeling, and Metal-driven rendering algorithms that ground 3D assets into the physical world.

Real-Time Light Estimation and Directionality

To render a shadow accurately, the rendering engine must first understand the lighting of the real environment. ARKit in iOS 12 analyzes the camera feed using machine learning to detect both the intensity and the primary direction of real-world light sources. Instead of relying solely on generic ambient light, ARKit 2 produces directional light vectors and spherical harmonics. This data determines where the virtual light source should be positioned in the 3D scene, matching the angle, color temperature, and brightness of the actual physical lighting in the room.

Invisible Shadow Receiver Planes

A major technical challenge in augmented reality is casting a shadow onto a real surface that already exists as a video feed. iOS 12 solves this by using plane detection to generate transparent 3D geometry—often referred to as a "shadow catcher" or "shadow receiver"—over detected flat surfaces like floors, tables, and countertops.

In frameworks like SceneKit and RealityKit, this geometry uses a specialized shader that renders only the shadow information while keeping the surface itself completely invisible. The shader multiplies the shadow values with the underlying camera pixel stream, darkening the real-world surface precisely where the virtual shadow falls without obscuring the texture of the actual floor or table.

Contact Shadows and Ambient Occlusion

To prevent virtual objects from appearing as though they are floating, iOS 12 incorporates contact shadows. These are small, sharp, localized drop-shadows generated specifically at the precise points where an object’s base meets a detected horizontal plane. By mimicking ambient occlusion—the natural blocking of diffuse ambient light in crevices and contact zones—these micro-shadows anchor the 3D asset firmly to the surface, regardless of the primary light source's direction.

Soft Shadows and Depth Calculations

Real-world shadows naturally exhibit penumbra: they are sharpest near the base of the object casting them and become progressively blurrier as the distance from the object increases. Using Apple’s low-level Metal graphics API, iOS 12 computes depth maps of virtual assets to produce Percentage-Closer Soft Shadows (PCSS). By calculating the distance between the virtual object, the virtual light source, and the detected physical surface, the system dynamically blurs the outer edges of the shadow, replicating authentic optical behavior in real time.