How iOS 15 Visual Look Up Recognizes Plants and Pets

Apple’s Visual Look Up feature in iOS 15 allows users to identify objects—specifically plants, flowers, dog breeds, and cat breeds—directly within the Photos app. This article explores the underlying technology powering this capability, explaining how Apple combines on-device machine learning, advanced computer vision, the Apple Neural Engine, and curated knowledge graphs to analyze and identify subjects instantly while maintaining user privacy.

On-Device Machine Learning and Computer Vision

Visual Look Up relies primarily on Apple’s Vision framework and Core ML running locally on the device. When an image is captured or viewed, the system executes computer vision algorithms to perform object detection and image segmentation. Instead of analyzing the entire photo as a flat file, the algorithm isolates the region containing the plant or animal, separating the subject from complex backgrounds like gardens, carpets, or outdoor environments.

Feature Extraction via Deep Neural Networks

Once the subject is isolated, deep convolutional neural networks (CNNs) extract high-dimensional visual features from the image. For pets, the network analyzes distinctive anatomical traits such as skull shape, ear structure, coat patterns, snout length, and fur texture. For plants, it scans leaf venation, petal arrangement, color gradients, and overall plant geometry. These visual characteristics are converted into compact mathematical representations known as feature vectors or embeddings.

Matching Against Curated Datasets

The generated embeddings are compared against a vast, pre-trained database of botanical and zoological classifications. Apple partnered with domain-specific data providers and trained its models on millions of labeled images covering thousands of common plant species, wild flora, and domestic pet breeds. The system measures the vector distance between the photo’s features and the reference database to find the closest statistical match. When a confidence threshold is met, the system assigns the most accurate species or breed name.

Integration with Siri Knowledge

After a match is verified, iOS connects the result to the Siri Knowledge Graph. When a user opens an eligible photo, the standard information icon (i) displays a small sparkle, indicating Visual Look Up has detected an item of interest. Tapping the icon presents an information card containing the identified breed or plant name, a brief overview from sources like Wikipedia, and related images from the web for verification.

Hardware Acceleration and Privacy

All initial image analysis, detection, and embedding generation happen on-device using the Apple Neural Engine (ANE) found in the A12 Bionic chip and later. Processing the imagery locally ensures rapid recognition speeds and prevents personal photos from being uploaded to external servers, ensuring that identifying personal pets and plants does not compromise user privacy.