How iOS 12 Tracks Animoji Tongue and Wink Movements

With the release of iOS 12, Apple enhanced its Animoji and Memoji features by adding the ability to detect winks and tongue protrusions in real time. This functionality is powered by the iPhone's TrueDepth camera system combined with advanced machine learning algorithms in Apple's ARKit platform. By projecting thousands of invisible infrared dots onto the face and reading depth discrepancies, iOS 12 maps minute facial muscle shifts and translates them directly into 3D avatar animations.

The TrueDepth Hardware Architecture

The tracking system relies on the TrueDepth camera cluster located in the notch of compatible devices. This setup consists of three main components working together:

  • Flood Illuminator: Bathes the user's face in invisible infrared (IR) light, allowing the system to see in varying lighting conditions, including total darkness.
  • Dot Projector: Projects over 30,000 distinct infrared dots onto the surface of the user’s face.
  • Infrared Camera: Captures the reflected IR pattern to generate a high-precision 3D depth map.

How iOS 12 Tracks Winks

In earlier versions of iOS, both eyes were often animated symmetrically or struggled with asymmetrical eyelid states. iOS 12 decoupled the tracking parameters for each eye using distinct ARKit blend shapes: eyeBlinkLeft and eyeBlinkRight.

When you wink, the TrueDepth sensor captures the deformation of the eyelid muscles and the sudden obstruction of the eyeball's reflective surface. The software processes the left and right eye regions independently. If the infrared camera records a closed eyelid on the left side while the right eye retains visible depth curvature and an open aperture, ARKit outputs a value closer to 1.0 (fully closed) for the left eye and 0.0 (open) for the right eye. The Memoji rig then mirrors this asymmetrical input.

How iOS 12 Tracks Tongue Movements

Detecting a tongue presented a unique engineering challenge because the inside of an open mouth typically registers as an empty depth void to an infrared camera. In iOS 11, opening the mouth simply caused the Animoji’s jaw to drop.

iOS 12 introduced the tongueOut blend shape using a combination of depth changes and computer vision:

  1. Depth Disruption: When your mouth is simply open, the IR light penetrates into the oral cavity, creating a low-reflectance, high-depth signature. When you stick out your tongue, the physical mass fills that cavity and projects forward, reflecting the IR dot pattern earlier than the back of the mouth would.
  2. Machine Learning Classification: Because teeth, lips, and tongues can all alter depth similarly, Apple trained a specialized neural network via the device’s Neural Engine. The model is trained to recognize the distinct shape, texture, and contour changes of an extruded tongue versus slightly parted lips or exposed front teeth.
  3. Real-Time Rigging: Once the neural network confirms tongue presence and calculates its protrusion distance, ARKit assigns a floating-point value to the tongueOut parameter. The graphics engine then triggers the 3D model’s tongue bone structure to slide forward through the avatar’s parted lips.

The Blend Shape Pipeline

The entire process operates at 60 frames per second. ARKit analyzes over 50 distinct facial blend shapes simultaneously—measuring brow tension, cheek puffs, jaw positions, and eyelid drops. By treating tongue protrusion and individual eye closures as discrete data points within this unified mesh, iOS 12 generates a fluid, natural representation of complex and playful facial expressions.