How Siri Fetches Definitions and Visual Look Up in iOS 16

In iOS 16, Siri and Visual Look Up retrieve definitions and contextual data through a hybrid pipeline combining on-device machine learning, local lexical databases, and Apple’s server-side Knowledge Graph. When a user requests a definition or taps a recognized subject in a photo, the system evaluates the request locally before querying privacy-preserving cloud services to deliver relevant, structured information without exposing raw personal data.

How Siri Fetches Web Definitions

When you ask Siri to define a word or concept in iOS 16, the system processes the request through several sequential stages:

  1. Natural Language Processing (NLP): Siri uses on-device speech-to-text and intent parsing to determine whether the query is a request for a dictionary definition, a factual concept, or a broader web query.
  2. Local System Dictionaries: For standard vocabulary, iOS 16 first consults its pre-installed reference dictionaries. If an exact match is found, the definition is displayed immediately without sending data to an external web index.
  3. Apple Knowledge Graph: If the query refers to an entity, pop-culture item, or specialized term not covered by standard dictionaries, Siri contacts Apple's Knowledge Graph. This is a massive database of structured entities sourced from Wikipedia, licensed databases, and verified open-web data.
  4. Web Crawling and Siri Suggested Websites: If the query cannot be resolved via dictionaries or the Knowledge Graph, Siri queries Apple’s web index (populated by Applebot). Instead of returning a list of links, Siri extracts structured snippets directly from high-authority web pages to present a concise definition box.

How Visual Look Up Operates

Visual Look Up in iOS 16 expands object recognition to pets, birds, insects, and statues, alongside existing categories like landmarks, plants, and artwork. The information retrieval process relies heavily on the Apple Neural Engine (ANE):

  1. On-Device Feature Detection: When a photo is viewed, the Vision framework runs deep convolutional neural networks directly on the device. It detects high-level subjects (such as a specific breed of dog or a flower) and highlights them with an interactive icon.
  2. Visual Embedding Generation: Rather than uploading the photo to the cloud, the device converts the detected visual features into a compact, mathematical vector (an embedding).
  3. Anonymous Cloud Lookup: The device sends this embedding, along with limited classification labels, to Apple’s Visual Search servers. The server matches the embedding against reference catalogs of known species, artworks, and landmarks.
  4. Contextual Enrichment: Once matched, the server returns structured reference cards containing encyclopedic definitions, taxonomies, and links to platforms like Wikipedia.

By keeping image processing and preliminary parsing entirely on-device, iOS 16 optimizes speed and preserves user privacy while fetching accurate definitions across both voice and visual interfaces.