PaliGemma: Google's Compact Vision-Language Model for Fine-Tuning

A VLM Built for Fine-Tuning

Most vision-language models are released with zero-shot capability as the headline feature. PaliGemma takes a different stance: it is designed as a transfer learning base. Google trained it to be fine-tuned on specific vision tasks rather than to be a general-purpose assistant out of the box. This design choice shapes everything about how you should use it.

Architecture: SigLIP + Gemma 2B

PaliGemma has two components:

1. SigLIP vision encoder - processes images at 224×224 (or 448×448 in the larger variant), producing 256 visual tokens via sigmoid loss contrastive pretraining (more stable than CLIP's softmax)
2. Gemma 2B language model - takes the visual tokens concatenated with text tokens and generates output autoregressively

Total parameter count is approximately 3B, making it practical on a single A100 or even high-end consumer GPUs with quantization.

What Fine-Tuning Tasks Look Like

PaliGemma was pretrained on a mixture of tasks using a unified text format. Fine-tuning for a new task follows the same pattern:

from transformers import PaliGemmaForConditionalGeneration, PaliGemmaProcessor
from PIL import Image
import torch

model_id = "google/paligemma-3b-pt-224"
processor = PaliGemmaProcessor.from_pretrained(model_id)
model = PaliGemmaForConditionalGeneration.from_pretrained(
    model_id, torch_dtype=torch.bfloat16, device_map="auto"
)

image = Image.open("chart.png")
prompt = "caption en"  # or "answer en What is the trend shown?"

inputs = processor(text=prompt, images=image, return_tensors="pt").to(model.device)
with torch.inference_mode():
    output = model.generate(**inputs, max_new_tokens=100)
print(processor.decode(output[0], skip_special_tokens=True))

Supported Fine-Tuning Tasks

PaliGemma's pretraining covered: image captioning, visual question answering, object detection (outputs bounding box coordinates in text), image segmentation (outputs segmentation tokens), OCR, and chart understanding. Fine-tuning on custom datasets for any of these tasks is straightforward because the model already has the relevant capabilities - you are adapting rather than teaching from scratch.

PaliGemma 2: The Updated Backbone

PaliGemma 2 replaced Gemma 2B with the Gemma 2 backbone and expanded to three sizes: 3B, 10B, and 28B. The 10B variant achieves strong performance on DocVQA (document visual QA) and ChartQA without task-specific engineering, making it the recommended starting point for enterprise document workflows.

When to Use PaliGemma vs GPT-4V

GPT-4V excels at general zero-shot visual reasoning but is expensive, opaque, and rate-limited. PaliGemma is better when you have labeled data for a specific task, need to run inference at scale on your own infrastructure, or require full control over the model weights for compliance reasons. For receipt parsing, form extraction, or domain-specific captioning, a fine-tuned PaliGemma 3B will typically outperform GPT-4V at a fraction of the cost.

Links

• HuggingFace model page
• PaliGemma paper
• HuggingFace fine-tune guide