CNNs, Vision Pipelines, and Transfer Learning

Convolutional Neural Networks (CNNs) show up across computer vision: image classification, object detection, face recognition, medical imaging, and visual search. If you work with image or spatial data, you need to know how they work and when pretrained models are enough.

Why Convolutions, Not Dense Layers?

Consider classifying a 224×224 RGB image. A fully connected layer would have 224 × 224 × 3 = 150,528 inputs. A single hidden layer with 1000 neurons requires 150 million parameters - most of which would overfit on any realistic dataset.

CNNs exploit two properties of images:

1. Local connectivity: nearby pixels are correlated; distant pixels usually aren't.
2. Translation invariance: a cat in the top-left corner and a cat in the bottom-right are both cats.

A convolutional layer applies a small filter (kernel) across the entire image, detecting the same local pattern everywhere. A 3×3 filter applied to a 150,528-input image uses only 27 parameters (plus bias) - shared across every spatial location.

How Convolution Works

pythonCopy
import torch
import torch.nn as nn
import torch.nn.functional as F

# A single conv layer
# in_channels=3 (RGB), out_channels=32 (32 filters), kernel_size=3
conv = nn.Conv2d(in_channels=3, out_channels=32, kernel_size=3, padding=1)
# padding=1 with kernel_size=3 preserves spatial dimensions

# For a batch of 8 images, 224x224 with 3 channels:
x = torch.randn(8, 3, 224, 224)
out = conv(x)
print(out.shape)  # [8, 32, 224, 224]  -  32 feature maps, same spatial size

# Pooling: downsamples spatial dimensions, increases receptive field
pool = nn.MaxPool2d(kernel_size=2, stride=2)
pooled = pool(out)
print(pooled.shape)  # [8, 32, 112, 112]

Each filter learns a different feature: one might detect horizontal edges, another vertical edges, another color gradients. Deeper layers compose these primitive features into increasingly complex patterns.

A Simple CNN Architecture

pythonCopy
class SimpleCNN(nn.Module):
    def __init__(self, num_classes=10):
        super().__init__()
        self.features = nn.Sequential(
            # Block 1: 3 → 32 channels, spatial 224 → 112
            nn.Conv2d(3, 32, kernel_size=3, padding=1),
            nn.BatchNorm2d(32),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(2, 2),

            # Block 2: 32 → 64 channels, spatial 112 → 56
            nn.Conv2d(32, 64, kernel_size=3, padding=1),
            nn.BatchNorm2d(64),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(2, 2),

            # Block 3: 64 → 128 channels, spatial 56 → 28
            nn.Conv2d(64, 128, kernel_size=3, padding=1),
            nn.BatchNorm2d(128),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(2, 2),
        )
        self.classifier = nn.Sequential(
            nn.AdaptiveAvgPool2d((1, 1)),  # global average pooling: [B, 128, 28, 28] → [B, 128, 1, 1]
            nn.Flatten(),
            nn.Linear(128, 256),
            nn.ReLU(inplace=True),
            nn.Dropout(0.5),
            nn.Linear(256, num_classes)
        )

    def forward(self, x):
        return self.classifier(self.features(x))

AdaptiveAvgPool2d((1, 1)) collapses each feature map to a single number (global average) - this makes the classifier head independent of input spatial size, which is standard in modern CNNs.

Transfer Learning: The Production Standard

Training a CNN from scratch requires millions of labeled images. For most production tasks, you have hundreds to thousands of examples. The solution: start from a pretrained model that already knows edges, textures, shapes, and object parts - then fine-tune the final layers on your data.

pythonCopy
import torchvision.models as models
from torchvision import transforms

# Load ResNet50 pretrained on ImageNet (1.2M images, 1000 classes)
model = models.resnet50(weights=models.ResNet50_Weights.DEFAULT)

# Option 1: Feature extraction - freeze all layers, train only final classifier
for param in model.parameters():
    param.requires_grad = False

# Replace the final fully-connected layer with your task's head
model.fc = nn.Linear(model.fc.in_features, num_classes)
# Only model.fc.parameters() will be updated

# Option 2: Fine-tuning - unfreeze later layers too
for param in model.layer4.parameters():
    param.requires_grad = True
for param in model.layer3.parameters():
    param.requires_grad = True

# Use a lower LR for pretrained layers, higher for new head
optimizer = torch.optim.AdamW([
    {'params': model.fc.parameters(), 'lr': 1e-3},
    {'params': model.layer4.parameters(), 'lr': 1e-4},
    {'params': model.layer3.parameters(), 'lr': 1e-5},
], weight_decay=1e-4)

Feature extraction (freeze all, train head): use when your dataset is very small (< 500 images) or very similar to ImageNet.

Full fine-tuning: use when your dataset is large enough (> 5K images) or quite different from ImageNet (medical scans, satellite imagery).

Data Augmentation

CNNs need diverse training examples to generalize. Data augmentation synthetically creates variation by applying random transformations to training images - each epoch sees a different version of each image.

pythonCopy
from torchvision.transforms import v2

train_transform = v2.Compose([
    v2.RandomResizedCrop(224, scale=(0.7, 1.0)),
    v2.RandomHorizontalFlip(p=0.5),
    v2.ColorJitter(brightness=0.3, contrast=0.3, saturation=0.2),
    v2.RandomRotation(degrees=15),
    v2.RandomGrayscale(p=0.1),
    v2.ToImage(),
    v2.ToDtype(torch.float32, scale=True),
    v2.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),  # ImageNet stats
])

val_transform = v2.Compose([
    v2.Resize(256),
    v2.CenterCrop(224),
    v2.ToImage(),
    v2.ToDtype(torch.float32, scale=True),
    v2.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
])

Critical: validation transform must not include random augmentation - only resize/crop/normalize. Augment only training data.

The ImageNet mean and std values are the standard normalization for any model pretrained on ImageNet.

Visualizing What the Network Learned

pythonCopy
import matplotlib.pyplot as plt
import numpy as np

# Visualize learned filters from the first conv layer
first_conv = list(model.children())[0]
filters = first_conv.weight.data.cpu().numpy()  # shape: [out_channels, 3, H, W]

fig, axes = plt.subplots(4, 8, figsize=(12, 6))
for i, ax in enumerate(axes.flat):
    if i >= filters.shape[0]:
        break
    f = filters[i]
    # Normalize to [0, 1] for visualization
    f = (f - f.min()) / (f.max() - f.min() + 1e-8)
    ax.imshow(np.transpose(f, (1, 2, 0)))
    ax.axis('off')

First-layer filters of pretrained models typically look like Gabor filters - oriented edge detectors. This visual sanity check confirms the model is learning meaningful features.

Common CNN Architectures and When to Use Them

Architecture	Parameters	When to Use
ResNet-50	25M	General-purpose baseline, well-understood
EfficientNet-B0/B4	5M / 19M	When parameter efficiency matters
ViT-B/16	86M	Large dataset (> 100K images), better accuracy
ConvNeXt-Tiny	28M	Modern CNN matching ViT, good with ImageNet pretrain
MobileNetV3	5M	Mobile / edge deployment

For most production tasks with limited data: ResNet-50 or EfficientNet-B4 pretrained on ImageNet.

Common Mistakes and Bad Instincts

Not normalizing with ImageNet mean/std when using a pretrained model. The pretrained model expects this specific normalization. Using different normalization or no normalization causes significantly worse performance.

Applying augmentation to the validation set. Validation performance should measure generalization, not the average of random augmented versions. Augment only training data.

Training all layers with the same learning rate after loading pretrained weights. Pretrained layers need a much lower learning rate (1e-5) than the new classification head (1e-3). Using the same LR will destroy the pretrained representations.

Using a model with 50M+ parameters on a dataset of 1,000 images. Heavy overfitting is guaranteed. Use a smaller model, aggressive dropout and augmentation, or freeze more layers.

Forgetting to call model.eval() before inference. BatchNorm uses running statistics in eval mode. In train mode it uses batch statistics, which causes incorrect predictions on single images.

Where to Go Next

• Module 19 (Recurrent Networks) covers sequence modeling - the parallel to CNNs for temporal and text data.
• Module 21 (Transfer Learning and Fine-Tuning) goes deeper on the practical workflow for adapting any pretrained model to a new task.
• The post neural-network-fundamentals covers the backpropagation mechanics that underlie both CNNs and RNNs.