Lecture 15 — Vision Transformers, Advanced Topics & Review

Lecture 15

Dr. Muhammad Kazim

Department of Intelligent Systems, University of Lahore

Agenda

Part A — Vision Transformers: attention, multi-head self-attention, positional encoding, ViT.
Part B — Self-supervised & multimodal: MAE, contrastive pretraining, CLIP, VLMs, open-vocabulary detection.
Part C — Frontiers: event cameras, a SLAM overview.
Part D — Responsible vision: ethics, privacy, dataset bias, safety & robustness.
Part E — Wrap-up: Lab 15 preview, key dates, final-exam coverage.

This is the last new-content lecture of the course; Week 16 is capstone demos and review only.

From CNNs to Transformers — the Bridge

CNNs (Lecture 13) build representations from a local, translation-equivariant prior: small kernels, weight sharing, hierarchical receptive fields.
Transformers, introduced for language (Vaswani et al., 2017), build representations from global self-attention: every token can attend to every other token from layer 1.
2020's Vision Transformer (ViT) showed the same attention mechanism works directly on image patches, with no convolutions at all.
Practical framing: CNNs bring a strong prior (good with little data); transformers bring flexibility (good with lots of data, or a pretrained backbone).

Attention: Intuition

For each element in a sequence, attention asks: "which other elements matter to me, and how much?"
Every token is projected into three vectors:
- Query (Q): "what am I looking for?"
- Key (K): "what do I contain?" (compared against queries)
- Value (V): "what do I actually pass along if attended to?"
A token's new representation is a weighted sum of Values, where the weights come from how well its Query matches every Key.

Scaled Dot-Product Attention

For query matrix \(Q\), key matrix \(K\), value matrix \(V\) (each a stack of row vectors, key/query dimension \(d_k\)):

\[ \mathrm{Attention}(Q,K,V) = \mathrm{softmax}\!\left(\frac{QK^{T}}{\sqrt{d_k}}\right)V \]

\(QK^{T}\) — a similarity score between every query and every key (a full \(n \times n\) attention matrix).
Divide by \(\sqrt{d_k}\) — keeps dot products from growing too large as \(d_k\) grows, which would push softmax into saturated, near-zero-gradient regions.
\(\mathrm{softmax}(\cdot)\) along each row — turns scores into a probability distribution over "who to attend to."
Multiply by \(V\) — produces the attended output as a weighted blend of Values.

Multi-Head Self-Attention

One attention "head" learns one notion of relevance. Real problems need several at once (e.g. one head for colour, one for shape, one for spatial proximity).
Multi-head attention runs \(h\) attention operations in parallel on learned linear projections of \(Q, K, V\), then concatenates and projects back:

\[ \mathrm{MultiHead}(Q,K,V) = \mathrm{Concat}(\mathrm{head}_1,\dots,\mathrm{head}_h)\,W^{O}, \quad \mathrm{head}_i = \mathrm{Attention}(QW_i^{Q}, KW_i^{K}, VW_i^{V}) \]

Each head operates in a smaller dimension \(d_k = d_{\text{model}}/h\), so total compute is comparable to one full-size head.
Self-attention means \(Q\), \(K\), \(V\) all come from the same sequence (image patches, in ViT).

Positional Encoding

Attention is permutation-invariant — it has no notion of order or position on its own.
Position must be injected explicitly, by adding a positional vector to each token embedding before the first layer.
The original Transformer uses fixed sinusoidal encodings:

\[ PE_{(pos,2i)} = \sin\!\left(\frac{pos}{10000^{2i/d_{\text{model}}}}\right), \qquad PE_{(pos,2i+1)} = \cos\!\left(\frac{pos}{10000^{2i/d_{\text{model}}}}\right) \]

ViT instead uses a simpler learned positional embedding per patch position — both work; learned is more common in modern vision transformers.

The Transformer Encoder Block

Multi-head self-attention, then a residual connection + LayerNorm.
A position-wise feed-forward network (two linear layers with a GELU/ReLU in between), then another residual + LayerNorm.
Stack \(L\) of these blocks to build depth.
Residual connections and pre/post-LayerNorm are what make 12–24+ layer transformers trainable at all.

x  --> [Multi-Head Attn] --> (+x) --> LayerNorm
   --> [Feed-Forward MLP] --> (+) --> LayerNorm --> out

PyTorch Sketch: Multi-Head Attention

import torch, torch.nn as nn

class MultiHeadSelfAttention(nn.Module):
    def __init__(self, d_model=768, n_heads=12):
        super().__init__()
        self.h, self.dk = n_heads, d_model // n_heads
        self.qkv = nn.Linear(d_model, 3 * d_model)
        self.proj = nn.Linear(d_model, d_model)

    def forward(self, x):                      # x: (B, N, d_model)
        B, N, D = x.shape
        qkv = self.qkv(x).reshape(B, N, 3, self.h, self.dk)
        q, k, v = qkv.permute(2, 0, 3, 1, 4)    # each (B, h, N, dk)
        attn = (q @ k.transpose(-2, -1)) / self.dk ** 0.5
        attn = attn.softmax(dim=-1)
        out = (attn @ v).transpose(1, 2).reshape(B, N, D)
        return self.proj(out)

This is exactly the block you will build (plus the encoder/decoder stack and masking) in Lab 15.

Lab 15 Preview — Transformer From Scratch

Read “Attention Is All You Need” (Vaswani et al., 2017) using the 5-step paper method from Lecture 13: skim → map the architecture → extract the maths → implement from scratch → reproduce & verify.
Implement, in PyTorch: scaled dot-product attention, multi-head attention, sinusoidal positional encoding, and the encoder/decoder stack (with causal masking on the decoder side).
Train on a small toy sequence task (e.g. copy or reverse a sequence, or a small translation pair) to verify the implementation end-to-end.
Deliverable: a working from-scratch Transformer + a paper summary that explicitly bridges the encoder block to ViT.

Vision Transformer (ViT): Patch Embedding

ViT turns an image into a sequence the same way a sentence is a sequence of tokens.
Split a \(H \times W\) image into fixed-size \(P \times P\) patches (typically 16×16), flatten each patch, and linearly project to \(d_{\text{model}}\):

\[ N = \frac{HW}{P^{2}} \quad \text{patches, each mapped to a } d_{\text{model}}\text{-dim embedding} \]

A 224×224 image with 16×16 patches produces \(N = 196\) tokens — the "sequence length" the transformer operates on.

ViT Architecture: [CLS] Token & Position Embeddings

Prepend a learnable [CLS] token to the patch sequence; its final-layer output is used for classification (borrowed from BERT).
Add a learned positional embedding to every patch token (including [CLS]) so the model knows patch order/location.
Feed the sequence through \(L\) standard transformer encoder blocks.
A linear classification head on the final [CLS] embedding produces class logits.

ViT vs CNN: Inductive Bias

Property	CNN	ViT
Locality prior	Strong (small kernels)	None built-in (learned from data)
Translation equivariance	Built-in	Only via learned position embeddings
Receptive field growth	Gradual, layer by layer	Global from layer 1
Data efficiency	Good on small/medium data	Needs large-scale pretraining to match CNNs
Compute pattern	Convolutions (local, cheap)	Full attention (quadratic in tokens)

Weaker inductive bias is a double-edged sword: worse from-scratch on small data, better once pretrained on enough of it.

Training ViT in Practice

The original ViT paper needed hundreds of millions of images (JFT-300M) to beat a well-tuned ResNet from scratch.
DeiT (Data-efficient Image Transformers) showed strong ViTs trainable on ImageNet-scale data alone, using aggressive augmentation and a distillation token that learns from a CNN teacher.
Practitioner takeaway: for a course-sized dataset, always start from an ImageNet-pretrained ViT (or a hybrid CNN+ViT) and fine-tune — never train ViT from scratch on a small dataset.

Self-Supervised Learning: Why Pretrain Without Labels

Labelled data is expensive; raw images/video are nearly free at scale.
Self-supervised learning (SSL) constructs a "pretext task" from the data itself (no human labels), then transfers the learned representation to a downstream task.
Two dominant families: masked prediction (reconstruct hidden parts of the input) and contrastive/self-distillation (pull augmented views of the same image together, push different images apart).

Masked Autoencoders (MAE)

Mask a large fraction of image patches (e.g. 75%); encode only the visible patches with a ViT encoder.
A lightweight decoder reconstructs the masked patches' raw pixels from the encoded visible tokens + mask tokens.
Pretext task: pixel reconstruction. Masking most of the image forces the encoder to learn holistic, semantic structure rather than copying local texture.
After pretraining, discard the decoder and fine-tune the encoder on the downstream task (classification, detection, segmentation).

Contrastive Self-Supervised Learning

SimCLR-style idea: take two random augmentations of the same image (crop, colour jitter, blur); train the encoder so their embeddings are close, while embeddings of other images in the batch are pushed apart (a contrastive/InfoNCE loss).
DINO-style idea: a student network predicts the output of a slowly-updated teacher network (no negative pairs needed); this self-distillation produces surprisingly strong, semantically meaningful attention maps in ViT.
Common thread: the supervisory signal comes from agreement between views, not from human labels.

CLIP: Contrastive Vision-Language Pretraining

Train an image encoder and a text encoder jointly on (image, caption) pairs scraped from the web (400M+ pairs in the original CLIP).
Loss: for a batch of \(N\) pairs, maximise cosine similarity between each image embedding and its matching text embedding, and minimise it against the other \(N-1\) mismatched pairs (a symmetric contrastive loss over the image–text similarity matrix):

\[ \mathrm{sim}(i, t) = \frac{f_{\text{img}}(i) \cdot f_{\text{txt}}(t)}{\lVert f_{\text{img}}(i) \rVert \, \lVert f_{\text{txt}}(t) \rVert} \]

Result: a shared embedding space where "a photo of a cat" lands near actual cat images — enabling zero-shot classification with no task-specific fine-tuning.

CLIP in Practice: Zero-Shot & Visual Search

from transformers import CLIPModel, CLIPProcessor
from PIL import Image

model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32")
proc  = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32")

image = Image.open("scene.jpg").convert("RGB")
labels = ["a cat", "a dog", "a drone"]
inputs = proc(text=labels, images=image, return_tensors="pt", padding=True)
probs = model(**inputs).logits_per_image.softmax(dim=1)[0]   # zero-shot scores

Same idea, run the other way, gives text→image visual search: encode a text query once, encode a folder of images, rank by cosine similarity (see 11_clip_visual_search.py).
Practitioner note: CLIP embeddings make a strong, label-free starting point for retrieval, clustering, and out-of-distribution flags in a deployed system.

Open-Vocabulary Detection & VLMs for Captioning/VQA

Open-vocabulary detection: replace a detector's fixed class head with a CLIP-style text embedding of arbitrary class names, so the model can localise categories it was never explicitly trained to detect (e.g. YOLOE-style visual/text prompting, mentioned back in Lecture 7’s YOLO family survey).
Vision-Language Models (VLMs) (e.g. BLIP, LLaVA-style models) combine a vision encoder with a language model to produce free-form captions and answer visual questions (VQA) about an image.
Applied pattern from the project bank: a VLM can act as a flexible "CCTV analyst" — answer natural-language questions about a scene without a bespoke detector for every possible query.

Event Cameras: How They Differ

A conventional camera captures a full frame at a fixed rate — every pixel is read out whether or not it changed.
An event camera has independent pixels that each fire an asynchronous "event" \((x, y, t, \text{polarity})\) the instant their log-brightness changes beyond a threshold.
Output is a sparse stream of events, not a sequence of frames — there is no fixed "frame rate" at all.

Event Cameras: Why They Matter

Microsecond-scale temporal resolution — no motion blur, ideal for very fast motion (drone racing, high-speed robotics).
Very high dynamic range (>120 dB) — usable in scenes conventional sensors clip (bright sun + deep shadow).
Low power, low data rate when the scene is static — only changing pixels produce data, which suits always-on edge deployment.
Trade-off: needs specialised (spiking/asynchronous) algorithms — standard CNN/ViT pipelines expect dense frames, so events are often first accumulated into pseudo-frames.

A SLAM Overview: The Problem

SLAM = Simultaneous Localization And Mapping: estimate a moving camera/robot's trajectory and build a map of the environment at the same time, with neither known in advance.
Core loop: predict motion → observe new landmarks/features → update both the pose estimate and the map → repeat.
Why it is hard: pose error and map error are coupled — a wrong pose corrupts the map, and a wrong map corrupts future pose estimates (compounding drift).

SLAM: Approaches & Loop Closure

Feature-based SLAM (e.g. ORB-SLAM): detect and match sparse keypoints (this course's Lecture 7–8 machinery) across frames, triangulate landmarks, and optimise pose + landmarks jointly (bundle adjustment).
Direct SLAM (e.g. DSO, LSD-SLAM): minimise photometric error directly on pixel intensities, skipping explicit feature extraction.
Loop closure: recognising a previously-visited place and correcting accumulated drift by closing the trajectory graph — without it, long runs drift unboundedly.
This course does not derive SLAM in full; treat this as the conceptual bridge from Lecture 10's calibration/stereo/epipolar geometry to a complete localisation-and-mapping robotics system.

Ethics, Privacy & Dataset Bias

Privacy: many of this course's project-bank systems (face recognition, licence-plate reading, crowd/loitering detection) process personally identifiable data — deployment requires consent, retention limits, and access control, not just technical correctness.
Dataset bias: a detector trained on daytime, single-demographic, or single-region imagery will silently under-perform outside that distribution — measure accuracy per subgroup, not only in aggregate.
Surveillance risk: the same detection/tracking/analytics pipeline that counts customers in a mall can be repurposed for mass surveillance — consider the deployment context, not only the algorithm.
Practitioner rule: document training-data provenance, known failure modes, and intended operating conditions for every model you ship.

Safety & Robustness in Deployed Vision Systems

Distribution shift: lighting, weather, camera angle, and sensor changes between training and deployment are the most common real-world failure cause — monitor confidence/behaviour drift in production.
Adversarial & edge-case inputs: occlusion, motion blur, glare, and (rarely, but worth knowing) crafted adversarial perturbations can flip a confident, wrong prediction.
Fail safe, not fail silent: a safety-relevant system (e.g. driver-drowsiness, accident detection) needs an explicit "I don't know" / low-confidence fallback, not a forced best guess.
Always report a confidence + operating envelope alongside a deployed model's accuracy number, as you will for the capstone.

From Perception to Action: A Drone Grounding Example

r = model.predict(frame, conf=0.4, classes=keep, verbose=False)[0]
if len(r.boxes):
    x0, y0, x1, y1 = largest_box(r.boxes)         # pick the target
    err = (x0 + x1) / 2 - frame_center_x
    yaw = int(0.15 * err) if abs(err) > 40 else 0  # proportional control
    tello.send_rc_control(0, 0, 0, yaw)            # act on the detection

This is the whole course in one loop: detect (Lectures 7–13) → decide (a simple proportional controller, no ML needed) → act (send a command to real hardware).
Note the deadband (abs(err) > 40) — a direct, practical instance of the "fail safe" principle: do nothing rather than jitter on noisy small errors.
The project bank's drone/robot themes (Appendix B of the applied-CV handbook) all follow this exact detect→decide→act shape.

Course Wrap-Up & Key Dates

This week (Week 15): Lab 15 — Transformer from scratch; project report due.
Week 16: capstone demos on Raspberry Pi 5 target hardware + short talk + Q&A + peer review; final-exam review.
Finals week: Final Exam (40 marks) — cumulative, weighted toward recognition, deep learning & embedded deployment (weeks 10–16).
Reminder: the Lab Project (capstone, 30 marks) is the single largest component of the course grade — prioritise it accordingly this week.

Final Exam Coverage

Cumulative, but weighted toward weeks 10–16:

Week(s)	Block
10	Calibration & Stereo
11	Optical Flow & Tracking
12	Classical Recognition & Neural Networks
13	CNNs, Segmentation Networks & Reading Research Papers
14	Edge Vision & the OpenCV 5.0 Deep Dive
15	Vision Transformers, Advanced Topics & Review
1–9	Lighter coverage: image formation through segmentation — know the definitions and the pipeline, not just the deep-learning half

Key Takeaways

Attention replaces convolution's local prior with a global, learned relevance mechanism — scaled dot-product attention plus multi-head projections plus positional encoding is the entire mechanism.
ViT applies this directly to image patches; it needs more data or pretraining than a CNN to reach the same accuracy, but scales further once pretrained.
Self-supervised pretraining (MAE, contrastive/DINO) and vision-language pretraining (CLIP, VLMs) are how modern vision systems get strong representations without hand-labelling everything.
Event cameras and SLAM extend the sensing and geometry ideas from earlier in the course to high-speed and long-duration autonomous operation.
Every deployed vision system carries ethical and robustness obligations that are part of the engineering, not an afterthought.