Neural Engine Role in iOS 12 Animoji Processing

This article explores how Apple's Neural Engine processes facial tracking data to power Animoji and Memoji in iOS 12. By offloading complex machine learning tasks from the main processor, the dedicated hardware processes sensor inputs in real time, translating subtle facial movements into 3D animations with low latency and minimal battery consumption.

The TrueDepth camera system captures facial geometry using an infrared camera, a flood illuminator, and a dot projector that projects over 30,000 invisible infrared dots onto the user's face. This setup generates a raw 3D mesh and infrared image of the user’s facial structure. The primary role of the Neural Engine—a dedicated machine learning hardware component within Apple's A11 and A12 Bionic chips—is to analyze this dense point-cloud data instantly.

Using trained neural networks, the Neural Engine converts raw depth data into mathematical representations of facial muscle movements, known as blend shapes. It tracks more than 50 distinct muscle movements across the eyes, brows, cheeks, nose, and mouth. Instead of relying on the general CPU or GPU, the Neural Engine executes these deep learning models locally at trillions of operations per second, ensuring the user's facial geometry is mapped continuously at high frame rates.

With the release of iOS 12, Apple expanded Animoji capabilities to include tongue detection and wink tracking. These additions required significantly more nuanced pattern recognition to distinguish subtle movements, such as a partially extended tongue or a slight eye squint, from baseline neutral expressions. The Neural Engine performs the real-time inference needed to classify these micro-expressions without introducing lag to the rendering pipeline.

Handling these machine learning calculations on dedicated hardware also preserves system efficiency. Because the Neural Engine specializes in matrix multiplication and parallel processing, it completes facial-recognition inferences using significantly less power than the main application processor. This keeps device thermals low and prevents battery drain during extended use in Messages or FaceTime calls.

Finally, the Neural Engine ensures user privacy. All facial tracking models and sensor calculations are processed entirely on-device. The raw depth data and resulting blend shapes are analyzed in isolated execution environments and mapped directly to the 3D Animoji character models without ever being transmitted to external cloud servers.