How Vision Extracts Subject Contours in iOS 16

In iOS 16, Apple introduced advanced subject-lifting capabilities that allow users and developers to isolate prominent objects from backgrounds seamlessly. This article details how Apple’s Vision framework achieves this process under the hood by combining on-device deep learning instance segmentation, pixel-level probability masks, and contour-tracing algorithms to create crisp, extractable vector paths in real time.

1. Neural Network-Based Instance Segmentation

The extraction process begins with machine learning models executed directly on the Apple Neural Engine (ANE). Rather than using traditional edge-detection filters, the Vision framework employs multi-task deep convolutional neural networks and vision transformers trained to identify salient foreground objects.

When an image is passed through an analysis request, the network classifies each pixel, categorizing instances such as people, pets, plants, and inanimate objects. This enables the system to recognize discrete semantic boundaries between the primary subject and complex background clutter, even in low-contrast conditions.

2. Generating the Soft Instance Mask

Once the subject is identified, the model outputs a high-resolution, single-channel floating-point mask (an alpha matte). In this mask:

  • Pixels with a value of 1.0 represent definite foreground.
  • Pixels with a value of 0.0 represent background.
  • Values between 0.0 and 1.0 represent ambiguous boundary zones, such as semi-transparent objects, hair, or fur.

This grayscale confidence map serves as the structural foundation for both pixel isolation and geometric boundary calculation.

3. Thresholding and Edge Detection

To convert a rasterized confidence mask into discrete contours, the Vision framework analyzes the mask's gradient changes. It applies an adaptive threshold across the probability matrix to determine the definitive cutoff line between subject and background.

Using algorithms conceptually derived from border-following techniques (such as the Suzuki-Abe method or marching squares), the framework traces along the threshold boundary. It locates continuous sequences of border pixels and organizes them into ordered topological loops representing outer boundaries and inner cutouts (holes).

4. Vectorization and Sub-Pixel Smoothing

A raw border trace produces a stepped, jagged pixel edge. Vision resolves this through vectorization and smoothing:

  • Sub-Pixel Interpolation: The framework uses the fractional values of the underlying alpha matte to interpolate the exact edge location between adjacent pixels, yielding sub-pixel accuracy.
  • Polygon Simplification: Algorithms such as the Ramer-Douglas-Peucker (RDP) algorithm reduce the number of redundant vertices along straight or gently curved lines.
  • Cubic Spline Approximation: The remaining vertices are converted into continuous Bézier curves, resulting in a smooth vector path (CGPath) that accurately mirrors the subject's silhouette.

5. Final Output and Rendering

The resulting path can be used directly for geometric clipping or returned as a vectorized contour for analysis. When rendering the visual "lift" effect, the system uses the vector path to bound the region while applying the soft alpha matte at the edges, ensuring that fine details like wisps of hair retain natural transparency while maintaining a sharp overall perimeter.