PaliGemma: Google's Compact Vision-Language Model for Fine-Tuning

Updated
7 min read
PaliGemma: Google's Compact Vision-Language Model for Fine-Tuning
TL;DR

PaliGemma combines SigLIP vision encoding with Gemma 2B language generation in a 3B model explicitly designed to be fine-tuned rather than used zero-shot.

A VLM Built for Fine-Tuning

Most vision-language models are released with zero-shot capability as the headline feature. PaliGemma takes a different stance: it is designed as a transfer learning base. Google trained it to be fine-tuned on specific vision tasks rather than to be a general-purpose assistant out of the box. This design choice shapes everything about how you should use it.

Architecture: SigLIP + Gemma 2B

PaliGemma has two components:

  1. SigLIP vision encoder - processes images at 224×224 (or 448×448 in the larger variant), producing 256 visual tokens via sigmoid loss contrastive pretraining (more stable than CLIP's softmax)
  2. Gemma 2B language model - takes the visual tokens concatenated with text tokens and generates output autoregressively

Total parameter count is approximately 3B, making it practical on a single A100 or even high-end consumer GPUs with quantization.

What Fine-Tuning Tasks Look Like

PaliGemma was pretrained on a mixture of tasks using a unified text format. Fine-tuning for a new task follows the same pattern:

python
from transformers import PaliGemmaForConditionalGeneration, PaliGemmaProcessor
from PIL import Image
import torch

model_id = "google/paligemma-3b-pt-224"
processor = PaliGemmaProcessor.from_pretrained(model_id)
model = PaliGemmaForConditionalGeneration.from_pretrained(
    model_id, torch_dtype=torch.bfloat16, device_map="auto"
)

image = Image.open("chart.png")
prompt = "caption en"  # or "answer en What is the trend shown?"

inputs = processor(text=prompt, images=image, return_tensors="pt").to(model.device)
with torch.inference_mode():
    output = model.generate(**inputs, max_new_tokens=100)
print(processor.decode(output[0], skip_special_tokens=True))

Supported Fine-Tuning Tasks

PaliGemma's pretraining covered: image captioning, visual question answering, object detection (outputs bounding box coordinates in text), image segmentation (outputs segmentation tokens), OCR, and chart understanding. Fine-tuning on custom datasets for any of these tasks is straightforward because the model already has the relevant capabilities - you are adapting rather than teaching from scratch.

PaliGemma 2: The Updated Backbone

PaliGemma 2 replaced Gemma 2B with the Gemma 2 backbone and expanded to three sizes: 3B, 10B, and 28B. The 10B variant achieves strong performance on DocVQA (document visual QA) and ChartQA without task-specific engineering, making it the recommended starting point for enterprise document workflows.

When to Use PaliGemma vs GPT-4V

GPT-4V excels at general zero-shot visual reasoning but is expensive, opaque, and rate-limited. PaliGemma is better when you have labeled data for a specific task, need to run inference at scale on your own infrastructure, or require full control over the model weights for compliance reasons. For receipt parsing, form extraction, or domain-specific captioning, a fine-tuned PaliGemma 3B will typically outperform GPT-4V at a fraction of the cost.

#paligemma#google#vision-language#fine-tuning#siglip

One AI engineering post, weekly

LLM benchmarks, prompt techniques, and token-cost breakdowns, delivered to your inbox.

What should we test or explain next?

Comments

Have a question or something to add? Sign in to join the discussion.

Mahmudul Haque Qudrati — CEO & ML Engineer at Pristren

Mahmudul Haque Qudrati

CEO & ML Engineer

Software architect, full-stack engineer, and machine learning practitioner. Founder & CEO of Pristren, CTO at Venture Kick-backed PIN Sport Application GmbH (Basel, Switzerland), Software Development Lead at DailyFrame GmbH (building Dimetra patient transport software), and creator of Zlyqor. Focused on custom workflow automation, scalable web and mobile applications, and production AI/ML systems.