Kosmos-2: Grounded Image Understanding That Links Text to Image Regions

Updated
7 min read
Kosmos-2: Grounded Image Understanding That Links Text to Image Regions
TL;DR

Microsoft's Kosmos-2 produces bounding box coordinates inline with its text output, connecting every noun and phrase in its response to a specific region of the image.

What Grounding Means for Vision-Language Models

Standard VQAs answer "what is in this image?" Standard VLMs describe what they see. Kosmos-2 goes further: it answers questions by linking specific phrases in its output to spatial coordinates in the image. Ask "where is the dog?" and the model responds with both a text description and the bounding box coordinates for the region it is describing.

The GRIT Dataset

Kosmos-2 was trained on GRIT (Grounded Image-Text pairs), a large-scale dataset of web images annotated with noun phrase / bounding box alignments. The model learns to produce <box> tokens inline with its text output, formatted as:

snippet
<p>The golden retriever</p><box>[[0.12, 0.34, 0.67, 0.89]]</box> is sitting next to <p>the red ball</p><box>[[0.71, 0.55, 0.84, 0.72]]</box>.

Coordinates are normalized to [0, 1] relative to image dimensions, making them resolution-independent.

Running Kosmos-2

python
from transformers import AutoProcessor, Kosmos2ForConditionalGeneration
from PIL import Image

model = Kosmos2ForConditionalGeneration.from_pretrained("microsoft/kosmos-2-patch14-224")
processor = AutoProcessor.from_pretrained("microsoft/kosmos-2-patch14-224")

image = Image.open("street_scene.jpg")
prompt = "<grounding>Describe this image in detail:"

inputs = processor(text=prompt, images=image, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=128)

generated_text = processor.batch_decode(outputs, skip_special_tokens=True)[0]
processed_text, entities = processor.post_process_generation(generated_text)

for entity_name, (start, end), bboxes in entities:
    print(f"Entity: {entity_name}")
    for bbox in bboxes:
        x1, y1, x2, y2 = bbox
        print(f"  Box: ({x1:.2f}, {y1:.2f}) -> ({x2:.2f}, {y2:.2f})")

Referring Expression Comprehension

Kosmos-2 supports the inverse task: given a natural language description of a region, locate it. "Find the person wearing a red hat" returns a bounding box - no object detection pipeline or separate grounding model needed.

Phrase Grounding in Captions

When generating captions, Kosmos-2 links each noun phrase to its corresponding region. This creates structured captions that are both human-readable and machine-parseable - useful for building image search indices, generating accessibility descriptions, or populating product databases from catalog images.

Practical Use Cases

The grounding capability unlocks applications that pure VQAs cannot handle: natural language object detection (find a specific object without pre-defined classes), visual evidence for answers (the model shows you which region justified its answer), and extracting structured data (all product prices from a receipt, with bounding boxes for validation).

Comparison to Standard VQA Models

BLIP-2 and LLaVA give accurate text answers but cannot tell you where in the image they found the information. Kosmos-2 trades some raw question-answering accuracy for spatial grounding - a worthwhile trade for applications where location matters.

#kosmos-2#microsoft#grounding#bounding-box#referring-expression

One AI engineering post, weekly

LLM benchmarks, prompt techniques, and token-cost breakdowns, delivered to your inbox.

What should we test or explain next?

Comments

Have a question or something to add? Sign in to join the discussion.

Mahmudul Haque Qudrati — CEO & ML Engineer at Pristren

Mahmudul Haque Qudrati

CEO & ML Engineer

Software architect, full-stack engineer, and machine learning practitioner. Founder & CEO of Pristren, CTO at Venture Kick-backed PIN Sport Application GmbH (Basel, Switzerland), Software Development Lead at DailyFrame GmbH (building Dimetra patient transport software), and creator of Zlyqor. Focused on custom workflow automation, scalable web and mobile applications, and production AI/ML systems.