Self-Distillation Without Labels

Standard vision model training requires millions of labeled images. DINOv2 from Meta demonstrates that self-supervised learning - using the model's own predictions as supervision - can produce features that surpass supervised training on downstream tasks.

DINOv2 uses self-distillation: a student network learns to match the outputs of a teacher network (exponential moving average of student weights). Both networks see different augmented views of the same image; the student must predict what the teacher sees for the global view. No labels needed.

LVD-142M Dataset

The quality of self-supervised learning depends heavily on training data diversity. DINOv2's LVD-142M (Large-scale Visual Deduplicated) dataset was curated through:

Starting with a large uncurated web image collection
Self-supervised retrieval to find images similar to curated reference datasets
Deduplication to remove near-duplicate images

This produces 142 million diverse, high-quality images without manual annotation - far more than ImageNet's 1.2 million.

Using DINOv2 as a Frozen Feature Extractor

import torch
from transformers import AutoImageProcessor, AutoModel
from PIL import Image

processor = AutoImageProcessor.from_pretrained("facebook/dinov2-large")
model = AutoModel.from_pretrained("facebook/dinov2-large")
model.eval()

image = Image.open("photo.jpg")
inputs = processor(images=image, return_tensors="pt")

with torch.no_grad():
    outputs = model(**inputs)

# CLS token: global image representation
cls_features = outputs.last_hidden_state[:, 0, :]  # [1, 1024]

# Patch tokens: spatial features for dense tasks
patch_features = outputs.last_hidden_state[:, 1:, :]  # [1, num_patches, 1024]

print(f"Global feature shape: {cls_features.shape}")
print(f"Patch feature shape: {patch_features.shape}")

Variant

Parameters

Dim

Speed

ViT-S/14

21M

384

Fastest

ViT-B/14

86M

768

Fast

ViT-L/14

307M

1024

Moderate

ViT-g/14

1.1B

1536

Slow

Fine-Tuning for Custom Vision Tasks

DINOv2 frozen features with a linear head achieve 86.5% top-1 on ImageNet - competitive with many fully supervised models. For custom classification:

import torch.nn as nn
from transformers import AutoModel

class DINOv2Classifier(nn.Module):
    def __init__(self, num_classes: int):
        super().__init__()
        self.backbone = AutoModel.from_pretrained("facebook/dinov2-base")
        self.classifier = nn.Linear(768, num_classes)
        # Freeze backbone for fast training
        for param in self.backbone.parameters():
            param.requires_grad = False

    def forward(self, pixel_values):
        outputs = self.backbone(pixel_values=pixel_values)
        cls = outputs.last_hidden_state[:, 0]
        return self.classifier(cls)

Training only the linear head converges in minutes and works well with as few as 50 labeled examples per class - DINOv2's features generalize to new visual domains without extensive fine-tuning.

The GitHub repository includes depth estimation and semantic segmentation examples using DINOv2 patch features directly.

DINOv2: Meta's Self-Supervised Vision Features That Beat Supervised Models

Self-Distillation Without Labels

LVD-142M Dataset

AI & ML insights, weekly

Mahmudul Haque Qudrati

Related Articles

ONNX: Export Any ML Model and Run It Anywhere

Gradient Descent Explained: How Machine Learning Models Actually Learn

Supervised Learning Explained: How Models Learn from Labeled Examples

Using DINOv2 as a Frozen Feature Extractor

ViT Backbone Variants

Fine-Tuning for Custom Vision Tasks

The workspace your team
actually needs

DINOv2: Meta's Self-Supervised Vision Features That Beat Supervised Models

Self-Distillation Without Labels

LVD-142M Dataset

AI & ML insights, weekly

Mahmudul Haque Qudrati

Related Articles

ONNX: Export Any ML Model and Run It Anywhere

Gradient Descent Explained: How Machine Learning Models Actually Learn

Supervised Learning Explained: How Models Learn from Labeled Examples

Using DINOv2 as a Frozen Feature Extractor

ViT Backbone Variants

Fine-Tuning for Custom Vision Tasks

The workspace your teamactually needs

The workspace your team
actually needs