Lecture 13 — CNNs, Segmentation Networks & Reading Research Papers

Lecture 13

Dr. Muhammad Kazim

Department of Intelligent Systems, University of Lahore

Agenda

Part A — CNN building blocks: convolution, pooling, receptive fields, batch norm.
Part B — Architecture lineage: LeNet → ResNet → MobileNet → EfficientNet; transfer learning.
Part C — Detection & segmentation networks: two-stage vs one-stage detectors, mAP; FCN/U-Net/DeepLab, Mask R-CNN.
Part D — How to read a research paper and reproduce an architecture from scratch, worked through this week's Lab 13 paper: U-Net.

Convolution, Revisited for Learning

Lecture 4 introduced convolution as a fixed hand-designed kernel (Gaussian, Sobel, …). A convolutional layer is the same sliding-window operation, except the kernel weights are learned by back-propagation instead of hand-designed.
A conv layer has C_in × C_out × k × k learnable weights, applied identically at every spatial location — this weight sharing is what makes CNNs far more parameter-efficient than a fully-connected layer on image-sized input.
Stacking many small (e.g. 3×3) conv layers lets the network build increasingly abstract features: edges → textures → parts → objects.

Padding & Stride

Padding adds a border of zeros so the kernel can be centred on border pixels — “same” padding keeps the output the same spatial size as the input.
Stride is how far the kernel moves between applications — stride 2 halves the output resolution and is a common alternative to pooling for down-sampling.
Output size for one spatial dimension: \[ O = \left\lfloor \frac{I + 2P - K}{S} \right\rfloor + 1 \] where \(I\) is input size, \(P\) padding, \(K\) kernel size, \(S\) stride.

Pooling

Max pooling: take the maximum value in each window — keeps the strongest activation, gives some translation invariance.
Average pooling: take the mean — smoother, sometimes used for the final global average pool that replaces a large fully-connected layer (used in ResNet, MobileNet, and the SmallCNN in Lecture 12's code).
Pooling (or a strided conv) reduces spatial resolution as depth increases, trading spatial precision for a larger receptive field and fewer downstream computations.

Receptive Fields

The receptive field of a unit is the region of the input image that can influence its value.
Stacking conv layers grows the receptive field additively per layer (roughly \(k-1\) pixels per 3×3 layer, before stride/pooling), while stride/pooling grows it multiplicatively.
Why it matters: a unit must "see" an object's full extent to recognise it — this is why deep networks with down-sampling can reason about large objects while a shallow network cannot, regardless of parameter count.

Parameter Counting

Conv layer parameters: \(C_{in} \times C_{out} \times k \times k\) (+ \(C_{out}\) biases) — independent of the input's spatial size.
Fully-connected layer on a flattened 128×128×64 feature map to 128 outputs: \(128\times128\times64\times128 \approx 134\) million parameters — this is why classification heads use global average pooling before the final FC layer, not a flatten.
Always compute this by hand once for a new architecture: it tells you whether the model will fit in your Pi 5's RAM before you try to load it.

Batch Normalisation

Normalises each channel's activations over the mini-batch to zero mean / unit variance, then applies a learned scale & shift: \[ \hat{x} = \frac{x-\mu_B}{\sqrt{\sigma_B^2+\epsilon}}, \qquad y = \gamma \hat{x} + \beta \]
Stabilises and accelerates training by keeping activation distributions consistent across layers as weights change ("internal covariate shift" — the original explanation; modern work attributes much of the benefit to a smoother loss landscape).
Acts as a mild regulariser (Lecture 12): batch statistics add noise similar in spirit to dropout.
Practitioner note: batch norm behaves differently in train() vs eval() mode in PyTorch — a frequent source of the "works in training, breaks at inference" bug.

CNN Architecture Lineage

1998            2012      2014        2015          2017/19
LeNet-5   →   AlexNet  →  VGG   →   ResNet   →  MobileNet / EfficientNet
digits        ImageNet    deeper    residual      mobile / scaled
(7 layers)    breakthrough (16-19)  connections    for the edge

Each step fixed a limitation of the previous one — trace why, not just what, because the same reasoning shows up whenever you pick a backbone for a new project.

LeNet-5 (1998)

LeCun's original CNN for handwritten-digit recognition (MNIST): conv → pool → conv → pool → fully-connected → output.
Established the conv+pool+FC recipe still used today, but at a scale (7 layers, ~60K parameters) suited to 32×32 grayscale digits.

AlexNet (2012)

Won ImageNet 2012 by a huge margin, kick-starting the deep-learning era in vision.
Key ingredients: ReLU instead of sigmoid/tanh (faster training, less vanishing gradient), dropout for regularisation, training on GPUs, and data augmentation.
8 layers, ~60M parameters — large by 2012 standards, modest today.

VGG (2014)

Showed that depth matters more than clever kernel shapes: stack many small 3×3 convolutions instead of a few large ones.
Two stacked 3×3 convs have the same receptive field as one 5×5 conv, but fewer parameters and an extra non-linearity — strictly more expressive for the same receptive field.
VGG-16/19 are simple and uniform but heavy (~138M parameters) — still a common choice as a feature extractor when simplicity beats efficiency.

ResNet (2015): Residual Connections

Problem: beyond a certain depth, plain networks got harder to train — gradients vanish/explode through many stacked layers, and accuracy degrades even on the training set.
Fix: a residual block learns a residual \(F(x)\) added to a skip connection: \[ y = F(x) + x \] — if the ideal mapping is close to identity, the block only has to learn a small correction, and gradients flow directly through the skip connection during back-propagation.
Enabled networks with 50, 101, even 152+ layers to train reliably. ResNet backbones (ResNet-18/34/50) remain a default choice in production pipelines today.

MobileNet: Depthwise-Separable Convolutions

A standard conv mixes spatial and cross-channel information in one step: cost \(k^2 \cdot C_{in} \cdot C_{out}\) per output pixel.
Depthwise-separable convolution splits this into (1) a depthwise 3×3 conv — one filter per input channel, spatial mixing only — followed by (2) a pointwise 1×1 conv that mixes channels: cost \(k^2 C_{in} + C_{in}C_{out}\), roughly \(8\text{–}9\times\) cheaper for typical \(k=3\).
This is exactly the efficiency trick that makes real-time CNN inference feasible on a Raspberry Pi CPU or a Hailo NPU without dedicated matrix hardware for large dense convolutions.

EfficientNet: Compound Scaling

Three ways to make a CNN bigger: more depth (layers), more width (channels), higher resolution (input size).
Scaling only one dimension hits diminishing returns quickly. EfficientNet scales all three together with a single compound coefficient \(\phi\), found by a small grid search, then applied uniformly (EfficientNet-B0 through B7).
Result: substantially better accuracy-per-FLOP than hand-scaled architectures — a good family to consider when you need to dial in an exact accuracy/latency operating point.

Transfer Learning

Feature extraction: freeze a pretrained backbone (e.g. ImageNet-trained ResNet/MobileNet), replace only the final classification head, and train just the head on your (small) dataset.
Fine-tuning: unfreeze some or all backbone layers and continue training at a small learning rate — usually gives a further accuracy bump once the head has stabilised.
Practical default: start frozen, train the head for a few epochs, then unfreeze the last few backbone blocks and fine-tune at \(\eta \approx 10^{-4}\text{–}10^{-5}\). This is exactly the workflow in this week's Lab 12 classifier and every YOLO custom-training run in Lab 12/14.

Two-Stage Detectors: R-CNN → Faster R-CNN

R-CNN (2014): generate ~2000 region proposals (selective search), run a CNN on each crop independently — accurate but painfully slow.
Fast R-CNN: run the CNN once on the whole image, then pool features for each proposed region (RoI pooling) — much faster, proposals still external.
Faster R-CNN: replaces external proposals with a learned Region Proposal Network (RPN) sharing the backbone — the whole pipeline is now trainable end-to-end.
Two stages (propose, then classify+refine) give strong accuracy at the cost of speed — a reasonable choice when latency budget is generous.

One-Stage Detectors: SSD & the YOLO Family

Skip the proposal stage entirely: predict class + box offsets directly from a dense grid of anchors (predefined box shapes) in a single forward pass.
SSD predicts from multiple feature-map scales for multi-scale detection in one shot.
YOLO (You Only Look Once) treats detection as one regression problem over a grid — dramatically faster, historically slightly less accurate than two-stage on small/crowded objects, though modern YOLO versions have closed most of that gap.
Post-processing: many overlapping boxes survive thresholding, so Non-Maximum Suppression (NMS) keeps only the highest-confidence box per object cluster (IoU-based suppression).
One-stage speed is exactly why YOLO is the course's default detector for Raspberry Pi deployment (Labs 12 & 14).

Evaluating Detectors: mAP

A detection is a true positive if its predicted box overlaps a ground-truth box with \(\text{IoU} \ge\) a threshold (commonly 0.5) and the class matches.
Sweep the confidence threshold to trace a precision–recall curve per class; Average Precision (AP) is the area under that curve.
mAP is AP averaged over all classes; mAP@0.5 uses a single IoU threshold, mAP@0.5:0.95 (the COCO metric) averages AP over IoU thresholds 0.5–0.95 in steps of 0.05 — a stricter, more localisation-sensitive score.

Semantic Segmentation: FCN & the Encoder–Decoder

Semantic segmentation assigns every pixel a class label (no notion of separate object instances).
FCN (Fully Convolutional Network, 2015): replace the classifier's final FC layers with 1×1 convs and upsample (transposed convolution) back to input resolution — the first end-to-end-trainable segmentation network.
General recipe since FCN: an encoder down-samples to build a semantic (but spatially coarse) representation; a decoder up-samples back to full resolution. The central design question is how to recover the fine spatial detail lost by the encoder's down-sampling — U-Net's answer is next.

U-Net: Skip Connections for Segmentation

U-Net (Ronneberger et al., 2015) is a symmetric encoder–decoder shaped like a "U": a contracting path (conv+pool, like a normal CNN) followed by an expanding path (upsample+conv) back to full resolution.
The key idea: at each decoder level, concatenate the matching encoder feature map (a "skip connection") before the next conv — this hands the decoder back the fine spatial detail the encoder threw away, which pure FCN-style upsampling could not recover well.
Originally built for biomedical image segmentation with very few training images; the architecture generalises far beyond that, and it is this week's from-scratch paper-implementation lab (Lab 13).

Instance Segmentation & DeepLab

Mask R-CNN: extends Faster R-CNN with a third head that predicts a per-instance binary mask alongside the class and box — gives instance segmentation (each object individually), not just semantic segmentation.
DeepLab: uses atrous/dilated convolutions (a kernel with gaps, growing the receptive field without losing resolution or adding parameters) and an ASPP (Atrous Spatial Pyramid Pooling) module to capture multi-scale context without the aggressive down-sampling other architectures rely on.
Bridge forward: Lab 12/14 also uses YOLO-seg, a one-stage detector extended with a segmentation head, trading some of Mask R-CNN's accuracy for real-time speed on the Pi.

How to Read a Research Paper (and Reproduce It)

Skim first — read the abstract, figures and conclusion to grasp the problem and the key idea before touching any equation.
Map the architecture — turn the model figure into a block diagram: inputs, layers, tensor shapes, and how blocks connect.
Extract the maths — write down the core equations (e.g. scaled dot-product attention, the segmentation loss) and exact layer dimensions.
Implement from scratch — build each block in PyTorch, checking tensor shapes at every step against the paper.
Reproduce & verify — train on a small dataset, compare behaviour/metrics to the paper, and note any gaps or simplifications.

This is the exact method you will apply twice this semester: to U-Net this week (Lab 13), and to the Transformer in Lecture 15 (Lab 15).

Worked Example: Applying the Method to U-Net

1. Skim: Ronneberger et al. 2015 — the problem is biomedical image segmentation with very few labelled training images; the key idea is symmetric encoder–decoder + skip connections.
2. Map the architecture: 4 down-sampling blocks (2× 3×3 conv + ReLU, then 2×2 max-pool, doubling channels each time) → bottleneck → 4 up-sampling blocks (2×2 up-conv, concatenate the matching encoder feature map, then 2× 3×3 conv) → final 1×1 conv to the number of classes.
3. Extract the maths: track the exact spatial size and channel count in and out of every block; the paper's original valid-padding convs shrink spatial size at every step, so record the crop needed to align encoder and decoder feature maps before concatenation (or use same-padding to sidestep it, a common, faithful simplification).
4–5. Implement & reproduce: build the double-conv block once and reuse it in both paths; verify every skip-connection concatenation's channel dimensions match before training; evaluate with IoU and Dice against a reference/library U-Net.

Code Skeleton: U-Net's Double-Conv Block

class DoubleConv(nn.Module):
    def __init__(self, in_ch, out_ch):
        super().__init__()
        self.block = nn.Sequential(
            nn.Conv2d(in_ch, out_ch, 3, padding=1), nn.BatchNorm2d(out_ch), nn.ReLU(),
            nn.Conv2d(out_ch, out_ch, 3, padding=1), nn.BatchNorm2d(out_ch), nn.ReLU())
    def forward(self, x):
        return self.block(x)

# Decoder step: upsample, concatenate the matching encoder feature map, then DoubleConv
class Up(nn.Module):
    def __init__(self, in_ch, out_ch):
        super().__init__()
        self.up = nn.ConvTranspose2d(in_ch, in_ch // 2, kernel_size=2, stride=2)
        self.conv = DoubleConv(in_ch, out_ch)
    def forward(self, x, skip):
        x = self.up(x)
        x = torch.cat([skip, x], dim=1)   # the skip connection
        return self.conv(x)

You will assemble the full encoder/bottleneck/decoder + IoU/Dice evaluation in Lab 13.

This Week’s Lab

Lab 13 — Paper Implementation I: U-Net from Scratch (PyTorch).

Read "U-Net: Convolutional Networks for Biomedical Image Segmentation" (Ronneberger et al., 2015).
Understand the contracting/expanding paths and skip connections; implement the full architecture from scratch in PyTorch.
Train on a small segmentation dataset; evaluate with IoU/Dice and compare against a library U-Net.

Deliverable: a from-scratch U-Net that segments a dataset + a one-page paper summary.

Key Takeaways

Convolution + pooling + depth builds a growing receptive field over increasingly abstract features; batch norm stabilises training as networks get deep.
Every major architecture advance (VGG, ResNet, MobileNet, EfficientNet) fixed a specific, nameable limitation of what came before — know the "why" so you can pick the right backbone for a new problem.
Two-stage detectors trade speed for accuracy; one-stage detectors (the YOLO family) are the course's default for real-time embedded deployment.
The 5-step paper-reading method turns "read a paper" into a concrete, repeatable engineering process — apply it to U-Net this week, and to the Transformer in Lecture 15.