ARKit 2 Simultaneous Face and World Tracking

ARKit 2 in iOS 12 does not support tracking human faces simultaneously with world tracking. Developers building augmented reality experiences in iOS 12 must choose between tracking the user's surrounding environment or tracking facial expressions, as the framework restricts sessions to a single active configuration at any given time.

The Configuration Limitation in ARKit 2

In ARKit 2, tracking modes are defined by mutually exclusive configuration objects. To track the physical environment, surfaces, and 3D objects, developers must run an ARWorldTrackingConfiguration, which utilizes the rear-facing camera. To track human faces, facial topology, and expressions, developers must run an ARFaceTrackingConfiguration, which relies exclusively on the front-facing TrueDepth camera system.

Because an ARSession can only run one configuration at a time, running face tracking automatically terminates world tracking, and vice versa.

Hardware and Pipeline Constraints

The primary reason for this limitation in iOS 12 is hardware and processing bandwidth. At the time of iOS 12, iOS devices did not support concurrent capture sessions from both the front TrueDepth camera and the rear wide-angle camera within ARKit. Simultaneous processing of dense 6DoF (six degrees of freedom) spatial mapping alongside high-frequency facial mesh generation exceeded the real-time resource allocation permitted by the framework.

Workarounds in iOS 12

Developers attempting to simulate both features in iOS 12 typically relied on two workarounds:

  • Session Switching: Apps would rapidly switch between ARWorldTrackingConfiguration and ARFaceTrackingConfiguration. However, this caused noticeable latency, visual stutter, and loss of real-time tracking continuity.
  • CoreML and Vision Framework: Developers ran an ARWorldTrackingConfiguration on the rear camera while using Apple's Vision framework to detect faces in 2D within the rear camera feed. While this allowed basic face detection in the world space, it did not provide TrueDepth-based 3D blend shapes or precise facial mesh tracking.

Resolution in Later Releases

Simultaneous face and world tracking became natively possible with the release of ARKit 3 on iOS 13. Apple introduced the userFaceTrackingEnabled property within ARWorldTrackingConfiguration, allowing devices equipped with an A12 Bionic chip or newer to track the user's face with the front camera while mapping the environment using the rear camera.