Lecture 11 — Optical Flow & Tracking

Lecture 11

Dr. Muhammad Kazim

Department of Intelligent Systems, University of Lahore

Today’s Roadmap

L21 — Optical Flow: brightness constancy & the aperture problem; Lucas–Kanade & Horn–Schunck; pyramidal KLT tracking; dense flow (Farnebäck).
L22 — Tracking: background subtraction (MOG2, KNN); the Kalman filter; mean-shift/CAMShift; correlation trackers (KCF, CSRT); tracking-by-detection & SORT; MOT metrics.

Reading: Szeliski 2e §9.3–9.4. Lab 11 this week: optical flow, OC-SORT/Strong-SORT, trajectory & speed, line-crossing counting. Assignment 3 out.

What Is Optical Flow?

Optical flow is the apparent 2-D motion field \((u, v)\) of brightness patterns between consecutive frames — our best image-space proxy for the true 3-D motion field, which we cannot observe directly from one camera.

Used for motion estimation, video stabilisation, action recognition, structure-from-motion initialisation, and (today) as the front end for lightweight tracking.
It is an estimate, not ground truth: uniform regions and specular surfaces can fool it badly.

Brightness Constancy & the Aperture Problem

Assumption: a point’s intensity does not change as it moves: \(I(x, y, t) = I(x+u, y+v, t+1)\). A first-order Taylor expansion gives the optical flow constraint equation:

\[ I_x u + I_y v + I_t = 0 \]

One equation, two unknowns \((u, v)\) per pixel — under-determined. This is the aperture problem: looking through a small window, only the flow component perpendicular to the local edge is observable; the along-edge component is invisible. Corners and textured patches constrain both components; a straight edge constrains only one.

Lucas–Kanade (Sparse, Local)

Assume flow is constant in a small window \(W\) (e.g. 5×5–15×15) around each tracked point, giving an over-determined least-squares system:

\[ \begin{bmatrix} \sum I_x^2 & \sum I_x I_y \\ \sum I_x I_y & \sum I_y^2 \end{bmatrix} \begin{bmatrix} u \\ v \end{bmatrix} = - \begin{bmatrix} \sum I_x I_t \\ \sum I_y I_t \end{bmatrix}

Solvable when the 2×2 structure matrix is well-conditioned — exactly the Harris “cornerness” condition from Lecture 7. Good features to track are good corners.
Fast, sparse (a handful of points), the workhorse of classical point tracking.

Horn–Schunck (Dense, Global)

Instead of a local window, Horn–Schunck estimates flow at every pixel by minimising a global energy that trades off data fidelity against a smoothness prior:

\[ E(u,v) = \iint \big(I_x u + I_y v + I_t\big)^2 + \alpha^2 \big(|\nabla u|^2 + |\nabla v|^2\big)\, dx\, dy \]

Produces a dense flow field even in textureless regions, by propagating information from edges via the smoothness term.
Blurs across true motion boundaries (over-smooths); slower than Lucas–Kanade; superseded in practice by variational methods (TV-L1) and Farnebäck, but foundational.

Pyramidal KLT Tracking

Lucas–Kanade alone only handles small motions (the Taylor expansion breaks down for large displacements). Pyramidal KLT fixes this:

Build a Gaussian pyramid of both frames.
Estimate flow at the coarsest level (small image → small effective motion).
Upsample the estimate as an initial guess for the next finer level; refine with Lucas–Kanade.
Repeat down to full resolution.

p0 = cv2.goodFeaturesToTrack(gray0, maxCorners=200, qualityLevel=0.3,
                              minDistance=7, blockSize=7)
p1, status, err = cv2.calcOpticalFlowPyrLK(
    gray0, gray1, p0, None, winSize=(15, 15), maxLevel=3,
    criteria=(cv2.TERM_CRITERIA_EPS | cv2.TERM_CRITERIA_COUNT, 10, 0.03))
good_new, good_old = p1[status == 1], p0[status == 1]

Dense Flow: Farnebäck

Farnebäck’s method estimates dense flow by locally approximating each neighbourhood with a quadratic polynomial and comparing the polynomial coefficients between frames to solve for displacement, at every pixel, in a pyramidal coarse-to-fine scheme.

flow = cv2.calcOpticalFlowFarneback(
    gray0, gray1, None, pyr_scale=0.5, levels=3, winsize=15,
    iterations=3, poly_n=5, poly_sigma=1.2, flags=0)
mag, ang = cv2.cartToPolar(flow[..., 0], flow[..., 1])

Dense flow is heavier than sparse KLT but gives a full motion field — useful for video compression, action recognition features, and motion segmentation.

From Flow to Tracking

Optical flow gives frame-to-frame motion; tracking maintains a persistent identity for an object across an entire video. Two broad families:

Motion/appearance trackers (background subtraction, Kalman, mean-shift, KCF/CSRT) — no detector required, follow a hand-off region continuously.
Tracking-by-detection (SORT and friends) — run a detector every frame, then associate detections across frames into tracks. This is what modern pipelines (and Lab 11/12) actually use.

Background Subtraction: MOG2 & KNN

Model each pixel’s history as a distribution of “background” values; a pixel far from that distribution is foreground.

MOG2 — each pixel is modelled as a mixture of Gaussians, updated online; adaptively decides the number of components and includes optional shadow detection.
KNN — a non-parametric k-nearest-neighbours model of recent pixel history; often more robust in dynamic backgrounds (moving leaves, water).

bgsub = cv2.createBackgroundSubtractorMOG2(history=500, varThreshold=16,
                                            detectShadows=True)
fg_mask = bgsub.apply(frame)                 # 255=fg, 127=shadow, 0=bg
fg_mask = cv2.morphologyEx(fg_mask, cv2.MORPH_OPEN, kernel)

The Kalman Filter

A recursive Bayesian estimator for a linear system with Gaussian noise — the classic engine behind smooth track prediction:

Predict: \(\hat{x}_k^- = A\hat{x}_{k-1}\), \(P_k^- = AP_{k-1}A^\top + Q\)

Update: \(K_k = P_k^- H^\top (HP_k^- H^\top + R)^{-1}\), \(\hat{x}_k = \hat{x}_k^- + K_k(z_k - H\hat{x}_k^-)\)

State \(x\) is typically \((c_x, c_y, \dot{c}_x, \dot{c}_y, w, h)\) — position, velocity, box size.
Predicts a box even when the detector misses a frame (occlusion), and smooths jittery detections. This is exactly the motion model inside SORT.

Mean-Shift & CAMShift Tracking

Mean-shift tracking: represent the target by a colour histogram; each frame, shift the search window toward the mode of the back-projected histogram (same mean-shift climb as Lecture 9’s segmentation, applied over time instead of over an image).
CAMShift (Continuously Adaptive Mean-Shift) additionally resizes and rotates the window based on the zeroth/first moments of the back-projection — handles scale change as the object approaches/recedes.
Cheap and fast, but drifts on similarly-coloured backgrounds and has no explicit motion model.

Correlation Trackers: KCF & CSRT

KCF (Kernelized Correlation Filters) — learns a discriminative correlation filter from the target patch; exploits the circulant structure of shifted patches to train and detect via the FFT, extremely fast.
CSRT (Channel and Spatial Reliability Tracking) — adds a spatial reliability map so the filter focuses on the actual object shape rather than its bounding box; more accurate, notably slower.

tracker = cv2.TrackerCSRT_create()
tracker.init(frame0, bbox)             # bbox = (x, y, w, h)
ok, bbox = tracker.update(frame1)      # every subsequent frame

Single-object, appearance-only — no re-detection if the target leaves and re-enters the frame.

Tracking-by-Detection & SORT

SORT (Simple Online and Realtime Tracking) is the minimal, still-standard recipe:

Run a detector every frame (e.g. YOLO) to get boxes.
Predict each existing track’s next box with a Kalman filter (constant-velocity model).
Associate predictions to new detections by IoU cost, solved optimally with the Hungarian algorithm.
Update matched tracks’ Kalman state; spawn new tracks for unmatched detections; delete tracks unmatched for too many frames.

Descendants add appearance re-identification to survive occlusion: DeepSORT (CNN embedding + IoU), ByteTrack (also associates low-confidence boxes, recovering more true positives), OC-SORT / BoT-SORT / Strong-SORT (better motion models and re-ID) — used in Lab 11/12.

Multi-Object Tracking (MOT) Metrics

MOTA (Multi-Object Tracking Accuracy) — combines false positives, misses, and identity switches into one number: \(1 - \frac{\text{FN}+\text{FP}+\text{IDsw}}{\text{GT}}\).
ID switches — how often a track’s identity flips onto the wrong object; the number that matters most for counting/analytics applications.
MOTP — average localisation precision of matched detections (how tight the boxes are, ignoring identity).
IDF1 — F1 score of identity-preserving matches over the whole track lifetime; increasingly preferred over MOTA for long-horizon evaluation.

Code — Tracking, Line-Crossing Count & Speed

from ultralytics import YOLO
model = YOLO("yolo26n.pt")

counted, last_cy = set(), {}
for r in model.track(source=0, tracker="bytetrack.yaml",
                      persist=True, stream=True):
    if r.boxes.id is None:
        continue
    for tid, (x0, y0, x1, y1) in zip(r.boxes.id.int().tolist(),
                                      r.boxes.xyxy.cpu().numpy()):
        cy = int((y0 + y1) / 2)
        if tid in last_cy and tid not in counted:
            if last_cy[tid] < LINE_Y <= cy:
                counted.add(tid)          # crossed downward -> count
        last_cy[tid] = cy

This is the pattern behind Lab 11’s line-crossing counter and speed estimator (speed from centre displacement / \(\Delta t\), calibrated to real units with Lecture 10’s intrinsics).

Common Pitfalls / Practitioner Notes

Optical flow assumes small inter-frame motion — fast objects or a low frame rate break Lucas–Kanade; use the pyramid, or increase frame rate.
Background subtractors need a warm-up period with no moving foreground to build an accurate model; a moving camera breaks the static-background assumption entirely.
SORT-family ID switches spike under occlusion and fast camera motion — re-ID embeddings (DeepSORT/Strong-SORT) trade FPS for identity stability.
Single-object trackers (KCF/CSRT) silently drift; always sanity-check the confidence/response score and re-initialise from a fresh detection periodically.
Report MOT metrics on a held-out clip, not the one you tuned thresholds on.

This Week’s Lab

Lab 11 — Motion & Tracking.

Optical flow (sparse KLT and/or dense Farnebäck) on live Pi video.
Tracking-by-detection with OC-SORT and Strong-SORT.
Object trajectory visualisation and speed estimation.
People/vehicle line-crossing counting.

Deliverable: a tracking + counting + speed application. Assignment 3 is out this week (due Week 12).

Key Takeaways

Optical flow rests on the brightness-constancy assumption; the aperture problem means only edge-perpendicular motion is locally observable.
Lucas–Kanade (sparse, local) and Horn–Schunck (dense, global) are the two classical poles; pyramids extend LK to large motions (KLT); Farnebäck gives dense flow efficiently.
Tracking-by-detection (SORT and its descendants: ByteTrack, OC-SORT, Strong-SORT) is today’s dominant paradigm: predict with Kalman, associate with the Hungarian algorithm, re-identify with appearance embeddings.
MOTA/IDF1/ID-switches are how you make tracking quality a number, not a vibe.