How iOS 16 Visual Look Up Recognizes Objects

Visual Look Up in iOS 16 expanded its machine learning capabilities to identify subjects such as birds, insects, and statues directly within the Photos app. This article explains the underlying technology behind this feature, detailing how Apple combines on-device computer vision, the Apple Neural Engine, deep learning classification models, and rich knowledge databases to instantly detect and identify specific subjects in your images.

Advanced On-Device Computer Vision

When a photo is taken or opened, iOS 16 runs computer vision algorithms entirely on the device. The system relies heavily on the Apple Neural Engine (ANE) embedded within Apple silicon chips to process visual data quickly and efficiently without draining battery life or sending raw images to external servers.

Semantic Segmentation and Subject Isolation

Before identifying what an object is, the system must locate it within the image. Visual Look Up uses semantic segmentation—the same technology that powers the iOS 16 "lift subject from background" feature—to differentiate foreground elements from background clutter. The algorithm traces the contours of distinct shapes, separating a bird perched on a branch, an insect on a leaf, or a statue in a town square from its surroundings.

Feature Extraction

Once isolated, the object is analyzed through deep convolutional neural networks (CNNs). The model extracts a multi-dimensional mathematical representation of the object's visual characteristics, known as a feature vector.

• Birds: The algorithm evaluates wing patterns, beak shapes, plumage colors, and body proportions.
• Insects: The system focuses on exoskeleton structures, antennae, leg configurations, and wing venation.
• Statues: The model scans for structural poses, sculpting mediums (such as bronze, marble, or stone), and surrounding architectural landmarks.

Classification and Knowledge Graph Matching

The generated feature vector is compared against a vast indexed catalog of known subjects. Apple trains its classification models on millions of curated, labeled images spanning thousands of biological species, famous artworks, and public monuments.

When the model identifies a high-confidence match between the photo’s features and a known class, the system activates the Visual Look Up icon (an "info" button with glowing stars). Tapping the icon links the recognized feature vector to Siri Knowledge, Wolfram Alpha data, or Wikipedia entries, presenting the user with the subject's common and scientific name, background history, and related web results.