Kosmos-2: Grounded Image Understanding That Links Text to Image Regions

What Grounding Means for Vision-Language Models

Standard VQAs answer "what is in this image?" Standard VLMs describe what they see. Kosmos-2 goes further: it answers questions by linking specific phrases in its output to spatial coordinates in the image. Ask "where is the dog?" and the model responds with both a text description and the bounding box coordinates for the region it is describing.

The GRIT Dataset

Kosmos-2 was trained on GRIT (Grounded Image-Text pairs), a large-scale dataset of web images annotated with noun phrase / bounding box alignments. The model learns to produce <box> tokens inline with its text output, formatted as:

<p>The golden retriever</p><box>[[0.12, 0.34, 0.67, 0.89]]</box> is sitting next to <p>the red ball</p><box>[[0.71, 0.55, 0.84, 0.72]]</box>.

Coordinates are normalized to [0, 1] relative to image dimensions, making them resolution-independent.

Running Kosmos-2

from transformers import AutoProcessor, Kosmos2ForConditionalGeneration
from PIL import Image

model = Kosmos2ForConditionalGeneration.from_pretrained("microsoft/kosmos-2-patch14-224")
processor = AutoProcessor.from_pretrained("microsoft/kosmos-2-patch14-224")

image = Image.open("street_scene.jpg")
prompt = "<grounding>Describe this image in detail:"

inputs = processor(text=prompt, images=image, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=128)

generated_text = processor.batch_decode(outputs, skip_special_tokens=True)[0]
processed_text, entities = processor.post_process_generation(generated_text)

for entity_name, (start, end), bboxes in entities:
    print(f"Entity: {entity_name}")
    for bbox in bboxes:
        x1, y1, x2, y2 = bbox
        print(f"  Box: ({x1:.2f}, {y1:.2f}) -> ({x2:.2f}, {y2:.2f})")

Referring Expression Comprehension

Kosmos-2 supports the inverse task: given a natural language description of a region, locate it. "Find the person wearing a red hat" returns a bounding box - no object detection pipeline or separate grounding model needed.

Phrase Grounding in Captions

When generating captions, Kosmos-2 links each noun phrase to its corresponding region. This creates structured captions that are both human-readable and machine-parseable - useful for building image search indices, generating accessibility descriptions, or populating product databases from catalog images.

Practical Use Cases

The grounding capability unlocks applications that pure VQAs cannot handle: natural language object detection (find a specific object without pre-defined classes), visual evidence for answers (the model shows you which region justified its answer), and extracting structured data (all product prices from a receipt, with bounding boxes for validation).

Comparison to Standard VQA Models

BLIP-2 and LLaVA give accurate text answers but cannot tell you where in the image they found the information. Kosmos-2 trades some raw question-answering accuracy for spatial grounding - a worthwhile trade for applications where location matters.

Links

• HuggingFace model page
• Kosmos-2 paper