Kosmos-2: Grounded Image Understanding That Links Text to Image Regions
Microsoft's Kosmos-2 produces bounding box coordinates inline with its text output, connecting every noun and phrase in its response to a specific region of the image.
Standard VQAs answer "what is in this image?" Standard VLMs describe what they see. Kosmos-2 goes further: it answers questions by linking specific phrases in its output to spatial coordinates in the image. Ask "where is the dog?" and the model responds with both a text description and the bounding box coordinates for the region it is describing.
The GRIT Dataset
Kosmos-2 was trained on GRIT (Grounded Image-Text pairs), a large-scale dataset of web images annotated with noun phrase / bounding box alignments. The model learns to produce <box> tokens inline with its text output, formatted as:
<p>The golden retriever</p><box>[[0.12, 0.34, 0.67, 0.89]]</box> is sitting next to <p>the red ball</p><box>[[0.71, 0.55, 0.84, 0.72]]</box>.
Coordinates are normalized to [0, 1] relative to image dimensions, making them resolution-independent.
Team workspace
Ship faster with chat, meetings, and projects in one place — Zlyqor.
from transformers import AutoProcessor, Kosmos2ForConditionalGeneration
from PIL import Image
model = Kosmos2ForConditionalGeneration.from_pretrained("microsoft/kosmos-2-patch14-224")
processor = AutoProcessor.from_pretrained("microsoft/kosmos-2-patch14-224")
image = Image.open("street_scene.jpg")
prompt = "<grounding>Describe this image in detail:"
inputs = processor(text=prompt, images=image, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=128)
generated_text = processor.batch_decode(outputs, skip_special_tokens=True)[0]
processed_text, entities = processor.post_process_generation(generated_text)
for entity_name, (start, end), bboxes in entities:
print(f"Entity: {entity_name}")
for bbox in bboxes:
x1, y1, x2, y2 = bbox
print(f" Box: ({x1:.2f}, {y1:.2f}) -> ({x2:.2f}, {y2:.2f})")
Referring Expression Comprehension
Kosmos-2 supports the inverse task: given a natural language description of a region, locate it. "Find the person wearing a red hat" returns a bounding box - no object detection pipeline or separate grounding model needed.
Phrase Grounding in Captions
When generating captions, Kosmos-2 links each noun phrase to its corresponding region. This creates structured captions that are both human-readable and machine-parseable - useful for building image search indices, generating accessibility descriptions, or populating product databases from catalog images.
Practical Use Cases
The grounding capability unlocks applications that pure VQAs cannot handle: natural language object detection (find a specific object without pre-defined classes), visual evidence for answers (the model shows you which region justified its answer), and extracting structured data (all product prices from a receipt, with bounding boxes for validation).
Comparison to Standard VQA Models
BLIP-2 and LLaVA give accurate text answers but cannot tell you where in the image they found the information. Kosmos-2 trades some raw question-answering accuracy for spatial grounding - a worthwhile trade for applications where location matters.
Practical deep-dives on LLMs, developer tools, and AI engineering. No filler. Unsubscribe any time.
// written byFIG. AUTH-01
530
Mahmudul Haque Qudrati
CEO & ML Engineer
CEO and ML Engineer at Pristren. Builds AI-powered software for teams and writes about machine learning, LLMs, developer tools, and practical AI applications.
ONNX (Open Neural Network Exchange) is the universal model format - export from PyTorch, scikit-learn, or HuggingFace and run 3x faster inference with ONNX Runtime on CPU or GPU.
