HuggingFace's Answer to GPT-4V
The HuggingFace Multimodal team (M4) released Idefics2 as a fully open alternative to closed multimodal APIs. At 8B parameters - Mistral 7B language backbone plus SigLIP vision encoder - it is deployable on a single A100 80GB and competitive with much larger proprietary models on several benchmarks.
Architecture Highlights
Idefics2 connects a SigLIP vision encoder to a Mistral 7B language model via a learned perceiver resampler. Unlike models that resize all images to a fixed resolution and pad the rest, Idefics2 preserves native aspect ratios by tiling: large images are split into sub-images that are encoded independently, then concatenated. This means a 1024×512 image and a 512×1024 image are both handled without distortion.
from transformers import AutoProcessor, AutoModelForVision2Seq
from PIL import Image
import torch
processor = AutoProcessor.from_pretrained("HuggingFaceM4/idefics2-8b")
model = AutoModelForVision2Seq.from_pretrained(
"HuggingFaceM4/idefics2-8b",
torch_dtype=torch.bfloat16,
device_map="auto",
)
image = Image.open("document.png")
messages = [
{
"role": "user",
"content": [
{"type": "image"},
{"type": "text", "text": "What is the total revenue shown in this table?"},
],
}
]
prompt = processor.apply_chat_template(messages, add_generation_prompt=True)
inputs = processor(text=prompt, images=[image], return_tensors="pt").to(model.device)
with torch.no_grad():
output = model.generate(**inputs, max_new_tokens=200)
print(processor.decode(output[0], skip_special_tokens=True))
