How iOS 12 Tracks Eye Blinking in Animoji

Apple’s iOS 12 introduced significant upgrades to Animoji and Memoji by adding nuanced facial movement detection, most notably independent eye blinking, winking, and tongue detection. This article explains how the underlying TrueDepth camera system, Apple's ARKit framework, and real-time machine learning algorithms work together to translate a user's physical eye blinks into digital character expressions with minimal latency.

The TrueDepth Camera System

The physical foundation of eye tracking in iOS 12 is Apple’s TrueDepth camera array. Rather than relying on a standard 2D video feed, the system utilizes specialized infrared components to measure depth and surface geometry:

  1. Flood Illuminator: Bathes the user's face in invisible infrared light, allowing the system to function in complete darkness or bright sunlight.
  2. Dot Projector: Beams more than 30,000 invisible infrared dots onto the face to map unique facial contours.
  3. Infrared Camera: Reads the dot pattern, captures the infrared image, and sends the raw data to the device's processor.

Because eyes and eyelids possess distinct contours, the infrared reflection changes shape significantly when an eyelid lowers. The TrueDepth camera captures these spatial depth changes at 60 frames per second, ensuring smooth tracking.

ARKit and Blend Shapes

The hardware data is processed through ARKit, Apple’s augmented reality framework. iOS 12 integrated an updated facial tracking system capable of mapping a face to a 3D mesh consisting of thousands of reference points.

To make Animoji mimic human movement, ARKit relies on "blend shapes"—a computer graphics technique where a base 3D model transforms based on specific structural movements. In the facial mesh dictionary, ARKit provides dedicated parameters for eye tracking, notably:

  • eyeBlinkLeft: Measures the closure of the left eyelid on a floating-point scale from 0.0 (completely open) to 1.0 (completely closed).
  • eyeBlinkRight: Measures the closure of the right eyelid on the same 0.0 to 1.0 scale.
  • eyeLookDown, eyeLookIn, eyeLookOut, eyeLookUp: Track gaze direction for each individual eye.

Because each eye has its own distinct blend shape coefficient, iOS 12 can track both eyes blinking synchronously or one eye closing independently during a wink.

Machine Learning and the Neural Engine

Raw depth maps alone are not always enough to differentiate between a partial squint, an involuntary twitch, or a deliberate blink. To solve this, iOS 12 uses onboard neural networks running on Apple's A-series Bionic Neural Engine.

The trained machine learning models analyze both the 2D infrared visual data and the 3D depth mesh simultaneously. This dual-input approach allows the device to recognize eyelid occlusion even if the user is wearing glasses or tilted at an angle. The Neural Engine calculates the blend shape coefficients in real time without offloading data to external servers.

Rendering the Animoji

Once the values for eyeBlinkLeft and eyeBlinkRight are calculated, ARKit passes them directly to the rendering engine powering the Animoji characters. The digital rigs use these values to deform the character's eye meshes proportionally. If a user slowly lowers their eyelids halfway, the Animoji’s eyelids match the exact position, creating a responsive and personalized expression.