CNNs or Domain Modeling Track: Vision / Time Series / Recsys

Software engineers entering ML often have a specific target domain: vision systems at an autonomous vehicle company, audio models at a speech startup, time-series forecasting at a fintech firm, graph models for fraud detection. This module covers CNNs as the canonical entry point into specialized domain architectures, and teaches you to read any domain-specific architecture using the general principles that underpin all of them.

Convolutional Neural Networks: The Core Idea

A CNN exploits spatial structure in data. Rather than learning a weight per pixel-per-hidden-unit (which would require billions of parameters for an image), a conv layer learns a small filter (e.g., 3×3) and applies it everywhere in the input via convolution. This gives two properties:

• Parameter sharing: the same 3×3 filter detects the same feature (edge, curve, texture) at any location
• Translation equivariance: if a cat moves two pixels right, the activations move two pixels right

pythonCopy
import torch
import torch.nn as nn

class SimpleCNN(nn.Module):
    def __init__(self, num_classes: int = 10):
        super().__init__()
        self.features = nn.Sequential(
            nn.Conv2d(3, 32, kernel_size=3, padding=1),   # (B, 32, H, W)
            nn.BatchNorm2d(32),
            nn.ReLU(),
            nn.MaxPool2d(2),                               # (B, 32, H/2, W/2)

            nn.Conv2d(32, 64, kernel_size=3, padding=1),  # (B, 64, H/2, W/2)
            nn.BatchNorm2d(64),
            nn.ReLU(),
            nn.MaxPool2d(2),                               # (B, 64, H/4, W/4)

            nn.Conv2d(64, 128, kernel_size=3, padding=1),
            nn.BatchNorm2d(128),
            nn.ReLU(),
            nn.AdaptiveAvgPool2d(1),                       # (B, 128, 1, 1)
        )
        self.classifier = nn.Linear(128, num_classes)

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        features = self.features(x).flatten(1)
        return self.classifier(features)

AdaptiveAvgPool2d(1) collapses spatial dimensions to a single value per channel regardless of input resolution. This makes the architecture resolution-agnostic.

Transfer Learning: Why You Almost Never Train from Scratch

For most vision tasks, training a CNN from scratch is the wrong starting point. ImageNet-pretrained models (ResNet, EfficientNet, ConvNeXt) already encode general visual features - edges, textures, shapes, object parts. Fine-tuning them requires orders of magnitude less data and compute.

pythonCopy
import torchvision.models as models

# Load EfficientNet-B0 pretrained on ImageNet
backbone = models.efficientnet_b0(weights="IMAGENET1K_V1")

# Replace the classifier head for your task
backbone.classifier = nn.Sequential(
    nn.Dropout(0.2),
    nn.Linear(backbone.classifier[1].in_features, num_classes),
)

# Freeze backbone initially - train only the head
for param in backbone.features.parameters():
    param.requires_grad = False

# After head converges, unfreeze and fine-tune with lower LR
# (discriminative learning rates: lower LR for early layers, higher for later)
optimizer = torch.optim.AdamW([
    {"params": backbone.features[:4].parameters(), "lr": 1e-5},
    {"params": backbone.features[4:].parameters(), "lr": 3e-5},
    {"params": backbone.classifier.parameters(), "lr": 1e-3},
], weight_decay=0.01)

Discriminative LR is the standard fine-tuning recipe: earlier layers encode generic features that need small adjustments; later layers and the head need larger updates.

Data Augmentation for Vision

Small training sets are the norm. Augmentation artificially expands the distribution the model sees:

pythonCopy
from torchvision import transforms

train_transform = transforms.Compose([
    transforms.RandomResizedCrop(224, scale=(0.7, 1.0)),
    transforms.RandomHorizontalFlip(),
    transforms.ColorJitter(brightness=0.2, contrast=0.2, saturation=0.1),
    transforms.RandomGrayscale(p=0.1),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
])

val_transform = transforms.Compose([
    transforms.Resize(256),
    transforms.CenterCrop(224),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
])

The Normalize mean/std values are the ImageNet statistics. Use them whenever you load an ImageNet-pretrained backbone - this ensures the input distribution matches what the pretrained model saw.

Domain Architecture Patterns: Reading Any Specialized Model

Every domain variant shares the same structural vocabulary. Once you can read a CNN, you can read any architecture:

1D convolutions for sequences and time series:

pythonCopy
# Same as 2D conv but kernel slides along time axis
nn.Conv1d(in_channels=1, out_channels=64, kernel_size=7, padding=3)
# Input: (batch, channels, time_steps)

Graph Convolutional Networks (GCNs) for graph-structured data:

The message-passing intuition: each node aggregates information from its neighbors, weighted by edge features. This generalizes convolution to arbitrary topology.

pythonCopy
# PyTorch Geometric (torch_geometric)
from torch_geometric.nn import GCNConv, global_mean_pool

class GCN(nn.Module):
    def __init__(self, in_channels, hidden, out_channels):
        super().__init__()
        self.conv1 = GCNConv(in_channels, hidden)
        self.conv2 = GCNConv(hidden, hidden)
        self.head = nn.Linear(hidden, out_channels)

    def forward(self, x, edge_index, batch):
        x = self.conv1(x, edge_index).relu()
        x = self.conv2(x, edge_index).relu()
        x = global_mean_pool(x, batch)  # aggregate nodes → graph embedding
        return self.head(x)

Point clouds / 3D geometry:

Point cloud models (PointNet, PointNet++) apply per-point MLPs then aggregate with max pooling. The key insight is permutation invariance - the output should not depend on the order you list the points.

Evaluating Vision Models

Beyond accuracy:

pythonCopy
from torchvision.ops import box_iou
from sklearn.metrics import average_precision_score

# Object detection: mAP (mean Average Precision) over IoU thresholds
# Image classification: top-1 and top-5 accuracy
# Segmentation: mean IoU (mIoU)

def topk_accuracy(outputs, targets, k=(1, 5)):
    with torch.no_grad():
        maxk = max(k)
        _, pred = outputs.topk(maxk, dim=1, largest=True, sorted=True)
        correct = pred.eq(targets.view(-1, 1).expand_as(pred))
        return {f"top{ki}": correct[:, :ki].any(dim=1).float().mean().item() for ki in k}

Common Mistakes and Bad Instincts

Not normalizing inputs with ImageNet statistics for pretrained models. A pretrained ResNet50 expects inputs normalized to the ImageNet mean/std. Feeding raw pixels or differently-normalized images degrades performance significantly - the backbone's learned features assume a specific input distribution.

Training from scratch on domain data < 50K samples. Below this rough threshold, a fine-tuned pretrained model almost always outperforms a randomly initialized one. The pretrained features are a strong prior.

Applying augmentations to the validation set. Augmentations are a training regularization technique. The validation set must see the clean, deterministic inference transform - otherwise your validation metric is noisy and optimistic.

Ignoring torch.utils.data.DataLoader prefetching. Vision data is large. Without num_workers > 0 and pin_memory=True, GPU utilization drops to 20–40% because the GPU waits for CPU data loading. These flags are free performance.

Where to Go Next

• transformers-and-modern-nlp-for-engineers: Transformers are now the dominant vision architecture too (ViT, DINOv2) - connect CNN intuitions to the attention mechanism
• fine-tuning-adaptation-and-when-not-to-fine-tune: the transfer learning techniques here generalize to every domain and model family

What to Practice Next

• Fine-tune a pretrained ResNet or EfficientNet on a small image dataset (e.g., Food-101 subset) using PyTorch Lightning, then visualize Grad-CAM activations to verify the model is focusing on the right regions.
• Implement a basic conv-pool block from scratch in PyTorch - define the layer sizes, run a forward pass on a random tensor, and verify the output spatial dimensions match your manual calculation.
• Identify a domain-specific dataset in your field of interest (medical imaging, satellite imagery, microscopy) and find a published baseline model; reproduce its reported metric on the official test split.