How TrueDepth Camera Enables Animoji in iOS 12

Apple’s TrueDepth camera system enables Animoji in iOS 12 by combining advanced infrared optical hardware with sophisticated real-time machine learning. Instead of relying on a standard 2D front-facing camera, the system captures precise three-dimensional geometry of the user's face. In iOS 12, this hardware interacts directly with updated ARKit frameworks and the A-series Bionic chip to translate over 50 distinct facial muscle movements—including newly added tongue and wink detection—directly onto animated 3D avatars with virtually zero latency.

The TrueDepth Hardware Architecture

The TrueDepth system is housed within the sensor notch at the top of the display and relies on three primary components working in unison:

  1. Flood Illuminator: Uses invisible infrared (IR) light to illuminate the user's face, ensuring the system functions accurately regardless of ambient lighting conditions, including complete darkness.
  2. Dot Projector: Beams more than 30,000 invisible infrared dots onto the contours of the user's face, creating an invisible, unique topographical grid.
  3. Infrared Camera: Reads the dot pattern, captures the reflected infrared image, and sends the raw data directly to the device's processor.

Depth Map Creation and Facial Tracking

Once the infrared camera captures the distortion of the projected dot grid, the system calculates the depth of every point based on how the pattern curves over facial features such as the nose, cheeks, eyes, and jawline. This data is transformed into a continuous 3D depth map of the user's face.

The Neural Engine inside the A11 or A12 Bionic processor analyzes this depth map in real time. Rather than merely detecting a static shape, it continuously tracks changes in the mesh at 60 frames per second.

iOS 12 Blend Shapes and Enhancements

In iOS 12, Apple expanded the facial tracking capabilities within ARKit, defining more than 50 distinct facial muscle expressions, known technically as "blend shapes." These include tracking for eyebrow raises, cheek puffs, jaw movement, and squinting.

iOS 12 introduced two major tracking milestones for Animoji and the newly introduced Memoji:

  • Tongue Detection: The infrared mesh gained the ability to recognize when the tongue protrudes from the mouth, mapping both its extension and direction.
  • Independent Eye Tracking and Winks: Improved coordinate mapping enabled the system to isolate one eye from the other, allowing users to wink and look around naturally.

Real-Time 3D Rig Mapping

The final stage of the process connects the tracked blend shapes to the Animoji character models. Each Animoji is built on a 3D digital skeleton rigged with proportional blend shapes that correspond directly to Apple's tracking framework.

When the TrueDepth sensor measures a 40% smile on the user's face, the software immediately applies a 40% value to the corresponding smile parameter on the 3D model. Because the depth calculations and rendering happen on-device using local hardware acceleration, the Animoji mirrors the user's real-life expressions, head tilt, and vocal cadence instantly without causing performance lag or draining excessive battery.