ARKit 2 3D Object Detection and Tracking in iOS 12

Apple’s ARKit 2, launched with iOS 12, fundamentally transformed mobile augmented reality by shifting the framework from simple 2D plane recognition to advanced 3D spatial awareness. This update introduced native capabilities to scan real-world objects, detect predefined physical items in real time, and anchor interactive digital experiences directly to them. By integrating persistent mapping, robust feature tracking, and multi-user synchronization, ARKit 2 significantly bridged the gap between virtual assets and physical 3D environments.

The Shift from 2D Surfaces to Real-World 3D Objects

Prior to ARKit 2, object recognition was largely restricted to flat horizontal and vertical surfaces or 2D image targets like posters and book covers. iOS 12 expanded this scope by enabling true three-dimensional awareness. Instead of merely identifying where a floor or wall existed, devices running ARKit 2 could analyze the physical volume, depth, and spatial geometry of distinct, non-flat items such as toys, sculptures, or museum artifacts.

The Scanning Workflow and ARReferenceObject

The foundation of 3D object detection in ARKit 2 begins with offline scanning and compilation:

  1. Object Scanning: Developers use an iOS-based scanning utility to capture a physical item from multiple angles. The camera records point-cloud feature maps representing the object's unique surface textures, contours, and physical dimensions.
  2. Feature Extraction: ARKit converts this point cloud into an ARReferenceObject, which contains the spatial footprint and identifiable visual anchors of the scanned item.
  3. Asset Packaging: These reference objects are embedded directly into the app bundle inside an asset catalog, enabling the device to search for them instantly without requiring constant internet access.

Real-Time Detection and ARObjectAnchor

During an active augmented reality session, ARKit 2 continuously compares the incoming camera frames and spatial data against the preloaded ARReferenceObject profiles.

When a match is found, the framework instantiates an ARObjectAnchor. This anchor represents the exact position and orientation (six degrees of freedom) of the physical object relative to the device. Developers can bind 3D models, visual effects, or informational UI directly to this anchor. Because the anchor tracks the object's specific orientation, virtual overlays align precisely with physical details, such as placing a digital label over a specific button on an appliance.

Improved Tracking Reliability

ARKit 2 enhanced tracking stability by fusing visual data from the camera with high-frequency readings from the device’s internal sensors (gyroscope and accelerometer). This sensor fusion minimizes drift when the user walks around a recognized object. Even if parts of the physical object become temporarily occluded or the user views it from steep angles, the tracking system maintains alignment far better than previous computer-vision models.

Persistence and Shared World Maps

Another major enhancement tied to 3D object detection is the introduction of ARWorldMap. In iOS 12, developers gained the ability to serialize the state of an AR session—including recognized 3D objects and their attached anchors—and save it to local storage or transmit it over a network.

This enables two core capabilities:

  • Session Persistence: A user can detect an object, place virtual content on it, leave the app, and return later to find the virtual content in the exact same physical position relative to the object.
  • Multi-User Collaboration: Multiple devices can share the same coordinate space, allowing different users to view and interact with the same 3D-anchored digital content simultaneously from their respective perspectives.