How ARKit 2 Tracks Vertical Surfaces in iOS 12

ARKit 2 in iOS 12 expanded augmented reality capabilities by introducing vertical plane detection alongside horizontal tracking. This article explains the technical mechanics behind how ARKit 2 identifies, tracks, and maps vertical surfaces such as walls, doors, and pillars. By combining camera sensor data with hardware motion tracking, feature point analysis, and dynamic boundary refinement, Apple enabled developers to place AR content seamlessly on upright structures.

Visual Inertial Odometry (VIO)

Vertical surface tracking begins with Visual Inertial Odometry (VIO). ARKit combines high-frequency motion data from the device’s Inertial Measurement Unit (IMU)—specifically the accelerometer and gyroscope—with visual information captured by the camera at 60 frames per second.

The IMU tracks the device's exact translation and rotation in three-dimensional space, providing real-time positioning. Concurrently, computer vision algorithms process camera frames to detect visual discrepancies. By fusing these two data streams, ARKit understands the camera’s perspective relative to the physical world, establishing the foundational coordinate system required to locate surfaces.

Feature Point Detection on Vertical Planes

To find a wall or door, ARKit scans the video feed for feature points—distinct, high-contrast visual elements such as surface textures, frame edges, wall art, or subtle changes in lighting.

When the user moves the device, ARKit tracks these feature points across consecutive frames. As the camera angle shifts, ARKit calculates the depth and distance of each point using parallax. Points that share a consistent perpendicular alignment relative to gravity are grouped together as potential vertical candidates.

Enabling Vertical Plane Detection

In iOS 12, vertical tracking is activated within an ARWorldTrackingConfiguration session by setting the planeDetection property:

let configuration = ARWorldTrackingConfiguration()
configuration.planeDetection = [.horizontal, .vertical]
sceneView.session.run(configuration)

With .vertical enabled, ARKit actively analyzes clusters of feature points to determine if they form a flat, upright 2D surface. The device’s gravity sensors ensure that the detected plane aligns with true vertical orientation.

Generating ARPlaneAnchor and Geometry

Once ARKit identifies a viable cluster of vertical points, it instantiates an ARPlaneAnchor with its alignment property set to .vertical.

ARKit 2 introduced refined plane geometry through the ARPlaneGeometry class. Instead of merely creating a generic rectangular bounding box, ARKit creates a rough polygonal mesh that mirrors the actual shape of the surface. As the user moves the device closer or pans across the wall, ARKit continuously collects more visual data, expanding the boundaries of the plane and refining its estimated center point and extent.

Handling Featureless Surfaces

Plain, untextured walls pose a major challenge for optical tracking because they lack distinctive feature points. ARKit 2 mitigates this by using the boundaries of the surface.

When facing a solid white wall, the system searches for adjacent anchors such as baseboards, door frames, light switches, ceiling intersections, and cast shadows. By tracking the edges where surfaces meet, ARKit can extrapolate the presence of the flat vertical plane between those boundaries, maintaining stable tracking even in minimally textured environments.