Training a deep neural network from scratch requires massive amounts of labeled data, significant compute, and weeks of experimentation. For most practical applications, you do not have any of those things. Transfer learning is the solution: start from a model that was pretrained on a large dataset, then adapt it to your specific task.
This is not a shortcut or a workaround. It is the standard approach for most real-world deep learning applications, and understanding it explains a large part of why modern AI systems work as well as they do.
The Core Insight: Lower Layers Are General
Deep neural networks learn hierarchical representations. In a convolutional network trained on images, the first layers learn to detect simple patterns: edges at various angles, color gradients, small textures. The middle layers combine these into more complex patterns: corners, curves, simple shapes. The final layers learn task-specific patterns: "this arrangement of features looks like a car" or "this arrangement looks like a dog."
The key observation: the lower, general layers are useful across many different tasks. A network trained to classify 1,000 ImageNet categories has learned to detect edges, textures, and shapes that are useful for classifying medical images, satellite imagery, manufacturing defects, or any other visual task.
This is why lower layers transfer. They encode general visual knowledge that is task-agnostic. Upper layers encode task-specific knowledge that must be replaced for a new task.
Two Transfer Learning Strategies
Feature extraction (frozen backbone): Load the pretrained model. Remove the final classification layer. Freeze all remaining layers (their weights will not be updated during training). Add new layers at the top appropriate for your task. Train only the new layers.
This approach is fast, requires little data, and is unlikely to damage the pretrained representations. Use it when your dataset is small (hundreds to low thousands of examples) or when your task is visually similar to the pretraining task.
from torchvision import models
import torch.nn as nn
model = models.resnet50(pretrained=True)
# Freeze all layers
for param in model.parameters():
param.requires_grad = False
# Replace the final layer for your number of classes
model.fc = nn.Linear(model.fc.in_features, num_classes)
# Only model.fc parameters will be updated during training
Fine-tuning: Load the pretrained model. Unfreeze some or all layers. Train the whole network (or just the unfrozen layers) on your task with a low learning rate.
Fine-tuning lets the model adjust its representations specifically for your task rather than using ImageNet representations verbatim. It typically achieves better final performance than pure feature extraction but requires more data (risk of overfitting if too little) and more careful training (risk of catastrophic forgetting if learning rate is too high).
The standard fine-tuning recipe:
- Start with a frozen backbone and train the new head until it converges.
- Unfreeze the last few pretrained layers and train with a very low learning rate (typically 10-100x lower than the head's learning rate).
- Optionally unfreeze more layers and repeat.
This graduated unfreezing helps avoid destroying the pretrained representations before the new head is stable enough to provide useful gradient signal.
