PyTorch Transfer Learning — Core Concepts

The Problem Transfer Learning Solves

Training a deep neural network from random weights requires three things most teams don’t have: massive datasets, expensive hardware, and weeks of training time. ResNet-50 was trained on 1.2 million ImageNet images using 8 GPUs for days. BERT was trained on 3.3 billion words of text. Few organizations can replicate that investment for each new project.

Transfer learning bypasses this by reusing models that have already been trained on large datasets. The pretrained weights encode general knowledge — visual features for vision models, language structure for text models — that transfers surprisingly well to new tasks.

Two Main Strategies

Feature Extraction

Freeze the pretrained model’s weights entirely. Replace only the final classification layer with one suited to your task. The pretrained layers act as a fixed feature extractor.

Best when: Your dataset is small (hundreds to low thousands of samples) and similar to the pretraining data.

Fine-Tuning

Start with pretrained weights, but allow some or all layers to update during training. Typically, you freeze early layers (which capture general features like edges and textures) and fine-tune later layers (which capture task-specific patterns).

Best when: Your dataset is medium-sized (thousands to tens of thousands) or your task differs significantly from the pretraining domain.

The Layer Hierarchy

In vision models trained on ImageNet, layers form a hierarchy:

Layer Depth	What It Learns	Transfer Value
Early (1-3)	Edges, colors, textures	Very high — universal across vision tasks
Middle (4-7)	Shapes, parts, patterns	High — useful for most domains
Late (8+)	Object-specific features	Medium — may need fine-tuning
Final classifier	ImageNet classes	None — always replaced

This hierarchy explains why transfer learning works: low-level visual features are essentially the same whether you’re classifying dogs or diagnosing X-rays.

Choosing a Pretrained Model

The model zoo has exploded. Key considerations:

• Domain match: A model pretrained on medical images transfers better to medical tasks than one trained on ImageNet
• Model size vs. your data: Larger models overfit faster on small datasets. EfficientNet-B0 often beats ResNet-152 when you have only 500 training images
• Architecture compatibility: Your task’s input format must match the model’s expectations (image size, number of channels)

Popular starting points: ResNet, EfficientNet, and Vision Transformers for images; BERT, RoBERTa, and GPT variants for text; Wav2Vec for audio.

Learning Rate Strategy

The most common mistake in transfer learning is using the same learning rate everywhere. Pretrained layers contain valuable knowledge that a high learning rate would destroy. Best practice:

• Final layer: Higher learning rate (e.g., 1e-3) — these weights are random and need fast learning
• Pretrained layers: Much lower learning rate (e.g., 1e-5) — gentle updates that refine without forgetting
• Frozen layers: Learning rate of zero — no updates at all

This “discriminative learning rate” approach, popularized by the fast.ai library, consistently outperforms uniform rates.

When Transfer Learning Fails

It’s not magic. Transfer learning underperforms when:

• Domains are too different. An ImageNet model transfers poorly to satellite imagery or microscopy without significant fine-tuning
• Your data has fundamentally different structure. A model trained on natural photos doesn’t understand spectrograms
• Negative transfer. Sometimes pretrained features actively hurt performance. If your task is very simple (binary classification on structured data), a small model from scratch may win

Common Misconception

People assume more pretraining data always means better transfer. This isn’t true — the relevance of pretraining data matters more than volume. A model pretrained on 100,000 domain-specific medical images often outperforms one pretrained on 14 million generic ImageNet images for medical tasks.

The one thing to remember: Transfer learning works because neural networks learn reusable features in early layers — borrowing those features and adapting only the top layers lets you build strong models with a fraction of the data and compute.