How iOS 18 Finds People by Activity in Photos

Apple’s iOS 18 introduces an upgraded Photos app powered by Apple Intelligence, enabling users to search their photo libraries using descriptive, everyday phrases. By combining advanced on-device computer vision, facial recognition, and large language models, the system can parse queries like "Dad grilling burgers in the backyard" or "Emma riding her bicycle" to locate precise moments. This article explains the underlying mechanisms iOS 18 uses to connect individual identity with specific activities.

Semantic Search and Vector Embeddings

Traditional photo search relies on matching basic keyword tags, such as "dog" or "beach." In iOS 18, the system utilizes multimodal semantic embeddings. The Apple Neural Engine translates both the visual content of a photograph and the natural language query into high-dimensional mathematical representations (vectors). When a search is performed, the system calculates the semantic similarity between the meaning of your text query and the contents of the image, allowing it to understand complex actions, poses, and interactions rather than relying on strict metadata matches.

Merging Identity with Action

To find a specific family member performing an activity, iOS 18 cross-references two separate machine learning models:

  1. Facial Recognition (People & Pets): iOS locally maintains facial signature clusters for individuals identified in your "People & Pets" album. When a name is entered in the search bar, the system retrieves the facial embedding associated with that specific person.
  2. Action and Object Detection: Concurrently, vision algorithms analyze the posture of the person, objects they are interacting with, and the environment (such as a bicycle, a spatula, or a swimming pool) to categorize the action.

The system then correlates the spatial coordinates of the identified family member with the detected action. For instance, in a group photo where multiple people are present, the algorithm ensures that the person matching the requested identity is the one actively engaging in the specified action, preventing false positives where one person is identified and a different person is performing the activity.

Natural Language Intent Parsing

Apple Intelligence enhances the search bar by functioning as an intent parser. When you type a query, the language model deconstructs the sentence into functional components:

  • Entities: Identifies the subject (e.g., "Sarah," "brother," "Grandma").
  • Actions/Verbs: Identifies the target behavior (e.g., "skateboarding," "sleeping," "reading").
  • Attributes and Context: Identifies supporting details like clothing color, location, or time of day (e.g., "in a red jacket," "at the park").

Once the intent is deconstructed, the system executes a compound search across the local database, filtering candidates until it finds the photos that satisfy both the identity and the activity criteria simultaneously.

On-Device Processing and Privacy

All index generation, facial recognition, and natural language interpretation occur locally on the device using Apple's Neural Engine. Photos and personal facial data are not uploaded to external servers to process these complex queries, ensuring that identity-based activity searches remain fully private.