Unpacking How Vision-Language Models Process Images Layer by Layer
A deep dive into how models like OpenAI CLIP, Google SigLIP, and Qwen2.5-VL handle images, from patch slicing to 3D M-RoPE and cross-modal attention.
Modern Vision-Language Models (VLMs) bridge the gap between visual inputs and natural language, but how they actually process images internally has long remained opaque. Recent empirical testing on a 96 GB NVIDIA RTX PRO 6000 Blackwell GPU gives a detailed look into the inner workings of models like OpenAI CLIP, Google SigLIP, and Qwen2.5-VL-3B-Instruct. By inspecting tensors, cosine similarities, positional IDs, and attention weights layer by layer, researchers have mapped out exactly how pixels transform into semantic understanding.
From Pixels to Patches and Tokens
The image-processing pipeline begins in the Vision Transformer (ViT). For a 560 × 560 RGB photograph containing 940,800 float values, a 14 × 14 pixel grid slices the image into 1,600 patches. A learned linear projection maps these raw RGB pixel values into 1,280-dimensional embedding vectors. To keep computation linear relative to image size rather than quadratic, Qwen2.5-VL uses 28 local-window ViT layers restricted to 112 × 112 pixel windows, alongside 4 global full-attention layers at layers 7, 15, 23, and 31.
Before entering the language model, a 2×2 Spatial Patch Merger compresses the visual data by a factor of four, reducing the 1,600 patches down to 400 visual word tokens at 2,048 dimensions each. This cuts quadratic self-attention pairs in the language model by 16 times, dropping them from 2.56 million down to 160,000 pairs. Meanwhile, models like Qwen2.5-VL employ 3D Multimodal RoPE (M-RoPE), partitioning rotary position frequencies into time, height, and width coordinates to enable a 1D language model to natively interpret 2D aspect ratios and 3D video timelines.
Why Typographic Attacks Fool Some Models
In 2021, researchers at OpenAI demonstrated that sticking an "iPod" label onto a green Granny Smith apple can cause OpenAI's CLIP to classify the fruit as an electronic music player. GPU measurements show that CLIP pools an entire image into a single 512-dimensional vector and multiplies raw image-text cosine similarities by a learned temperature scale of 100.0 before applying Softmax. Because written text strongly predicts web captions, the OCR features from the sticker nudge the iPod cosine similarity slightly above the green apple score, which the steep Softmax scaling turns into a decisive victory.
In contrast, modern architectures handle this differently. Google SigLIP replaces batch Softmax with an independent pairwise Sigmoid function, treating every image-text pair as a standalone binary classification problem with a learned scale of 117.33 and a bias of -12.93. When tested on the same labeled apple photographs, SigLIP successfully maintains its classification of the fruit. Furthermore, modern VLMs retain hundreds of individual spatial patch tokens rather than a single bottleneck vector, allowing the language model to attend separately to the physical fruit and the printed text.
What it means for developers
Understanding these internal mechanisms helps engineers choose the right model architectures for tasks like document OCR, spatial grounding, and multimodal reasoning. Native dynamic resolution approaches—such as NaViT—allow models to ingest images at their native aspect ratios, preventing fine text from blurring on wide documents or manifests. Developers looking to experiment with these capabilities can try top AI models cheaply through one API at https://apixoai.online.
Additionally, telemetry across the 36 layers of Qwen2.5-VL-3B reveals how middle-to-upper layers (specifically layers 17 to 28) house cross-modal grounding heads. These heads can concentrate up to 96.8% to 99.8% of their visual attention mass onto a target object's specific quadrant, explaining how text tokens successfully link to corresponding visual regions during inference.
Source: How Vision-Language Models Actually See Images: Inside ViT, CLIP, SigLIP & 3D M-RoPE — Towards AI. Written by the Apixo team from that report.
One key for Claude, GPT, GLM, DeepSeek and more. Pay per token with crypto.
Get your API key
