How iOS 15 Live Text Detects Non-Latin Scripts

Apple introduced Live Text in iOS 15 to allow users to interact with text inside photos, screenshots, and live camera feeds. This article explores the underlying technology that powers Live Text, focusing on how on-device machine learning, deep neural networks, and Apple's Vision framework identify, segment, and recognize complex non-Latin characters, such as Traditional and Simplified Chinese, with speed and precision.

On-Device Neural Processing

Live Text relies entirely on on-device intelligence powered by the Apple Neural Engine (ANE) integrated into Apple Silicon (A12 Bionic and later). By keeping processing on the device, iOS 15 processes visual data in real time with low latency while maintaining user privacy.

Text Detection and Region Proposal

Before reading individual characters, the system must locate text in an unconstrained visual environment. Live Text employs a specialized convolutional neural network (CNN) within Apple’s Vision framework to scan the entire image for text-like patterns. This network generates bounding boxes around regions containing text, regardless of whether the glyphs are Latin letters, numbers, or complex non-Latin logograms. The detector evaluates contrast, edge patterns, and spatial consistency to separate text from background textures.

Script Classification and Character Segmentation

Once text regions are defined, the system determines the script type. Non-Latin scripts, such as Chinese, present unique challenges due to dense stroke counts, square aspect ratios, and the absence of whitespace between words.

  1. Feature Extraction: A deep neural network extracts visual features from the bounded area, identifying geometric shapes, line intersections, and stroke radicals.
  2. Script Identification: Based on these extracted features, the system classifies the language family to route the data to the appropriate specialized optical character recognition (OCR) model.

Optical Character Recognition (OCR) for Logograms

For Latin text, OCR models typically recognize a small alphabet of roughly 26 letters in upper and lower cases. Non-Latin scripts, particularly Chinese, require recognizing thousands of unique characters. Live Text addresses this using deep sequence models:

  • Radical and Stroke Decomposition: Rather than treating every Chinese character as an entirely separate class, the model learns the visual hierarchy of sub-components (radicals) and strokes.
  • Connectionist Temporal Classification (CTC) / Attention Mechanisms: The model maps visual sequences directly into character predictions, accurately decoding characters even when they appear in stylized fonts, handwritten formats, or varied orientations.

Contextual Disambiguation via Natural Language Processing

Visual recognition alone is often insufficient for non-Latin scripts, where visually identical strokes can form different characters depending on context. Live Text integrates lightweight, on-device language models that evaluate character co-occurrence probabilities. By analyzing the surrounding text, the system resolves ambiguities, ensuring that the final output matches natural phrasing and grammatical structures.

Post-Processing and Interaction

After character recognition, the system aligns the recognized text with the image's coordinate space. This creates an invisible, interactive text layer over the image, allowing users to highlight, copy, translate, or look up non-Latin characters seamlessly in the Photos and Camera apps.