PaliGemma: Google's Compact Vision-Language Model for Fine-Tuning
PaliGemma combines SigLIP vision encoding with Gemma 2B language generation in a 3B model explicitly designed to be fine-tuned rather than used zero-shot.
Most vision-language models are released with zero-shot capability as the headline feature. PaliGemma takes a different stance: it is designed as a transfer learning base. Google trained it to be fine-tuned on specific vision tasks rather than to be a general-purpose assistant out of the box. This design choice shapes everything about how you should use it.
Architecture: SigLIP + Gemma 2B
PaliGemma has two components:
SigLIP vision encoder - processes images at 224×224 (or 448×448 in the larger variant), producing 256 visual tokens via sigmoid loss contrastive pretraining (more stable than CLIP's softmax)
Gemma 2B language model - takes the visual tokens concatenated with text tokens and generates output autoregressively
Total parameter count is approximately 3B, making it practical on a single A100 or even high-end consumer GPUs with quantization.
Team workspace
Ship faster with chat, meetings, and projects in one place — Zlyqor.
PaliGemma was pretrained on a mixture of tasks using a unified text format. Fine-tuning for a new task follows the same pattern:
from transformers import PaliGemmaForConditionalGeneration, PaliGemmaProcessor
from PIL import Image
import torch
model_id = "google/paligemma-3b-pt-224"
processor = PaliGemmaProcessor.from_pretrained(model_id)
model = PaliGemmaForConditionalGeneration.from_pretrained(
model_id, torch_dtype=torch.bfloat16, device_map="auto"
)
image = Image.open("chart.png")
prompt = "caption en" # or "answer en What is the trend shown?"
inputs = processor(text=prompt, images=image, return_tensors="pt").to(model.device)
with torch.inference_mode():
output = model.generate(**inputs, max_new_tokens=100)
print(processor.decode(output[0], skip_special_tokens=True))
Supported Fine-Tuning Tasks
PaliGemma's pretraining covered: image captioning, visual question answering, object detection (outputs bounding box coordinates in text), image segmentation (outputs segmentation tokens), OCR, and chart understanding. Fine-tuning on custom datasets for any of these tasks is straightforward because the model already has the relevant capabilities - you are adapting rather than teaching from scratch.
PaliGemma 2: The Updated Backbone
PaliGemma 2 replaced Gemma 2B with the Gemma 2 backbone and expanded to three sizes: 3B, 10B, and 28B. The 10B variant achieves strong performance on DocVQA (document visual QA) and ChartQA without task-specific engineering, making it the recommended starting point for enterprise document workflows.
When to Use PaliGemma vs GPT-4V
GPT-4V excels at general zero-shot visual reasoning but is expensive, opaque, and rate-limited. PaliGemma is better when you have labeled data for a specific task, need to run inference at scale on your own infrastructure, or require full control over the model weights for compliance reasons. For receipt parsing, form extraction, or domain-specific captioning, a fine-tuned PaliGemma 3B will typically outperform GPT-4V at a fraction of the cost.
Practical deep-dives on LLMs, developer tools, and AI engineering. No filler. Unsubscribe any time.
// written byFIG. AUTH-01
530
Mahmudul Haque Qudrati
CEO & ML Engineer
CEO and ML Engineer at Pristren. Builds AI-powered software for teams and writes about machine learning, LLMs, developer tools, and practical AI applications.
Supervised learning is the most widely used ML paradigm. Here is exactly how the train-measure-adjust loop works, where labels come from, and when the approach breaks down.
