Lecture 9
Department of Intelligent Systems, University of Lahore
Reading: Szeliski 2e §7.5. Lab 9 this week; the project proposal is due.
Segmentation partitions an image into regions that are homogeneous with respect to some property (colour, texture, motion) and, ideally, correspond to meaningful scene structures.
Two flavours you will meet later with deep nets: semantic segmentation (label per class, instances merged) and instance segmentation (label per object instance). Today is entirely classical, pixel- and region-based.
Pros: simple, connects naturally, good with a good seed. Cons: seed-dependent, sensitive to noise, slow on large images (implement with a queue/stack, not recursion).
Homogeneity test: e.g. \(\max(I) - \min(I) < T\) or variance < threshold. Produces a quadtree; naturally multi-resolution and less seed-dependent than region growing, but blocky boundaries (mitigated by the merge step).
Treat every pixel as a point \(x_i \in \mathbb{R}^3\) (e.g. Lab or HSV) and cluster into \(K\) groups minimising:
\[ J = \sum_{k=1}^{K} \sum_{x_i \in C_k} \lVert x_i - \mu_k \rVert^2 \]
Practitioner note: cluster in Lab, not RGB — perceptual distance behaves much better; add \((x, y)\) to the feature vector for spatial coherence.
import cv2, numpy as np
img = cv2.imread("scene.jpg")
lab = cv2.cvtColor(img, cv2.COLOR_BGR2LAB)
Z = lab.reshape((-1, 3)).astype(np.float32)
K = 4
criteria = (cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 20, 1.0)
_, labels, centers = cv2.kmeans(Z, K, None, criteria, 5, cv2.KMEANS_PP_CENTERS)
centers = centers.astype(np.uint8)
seg = centers[labels.flatten()].reshape(img.shape)
seg_bgr = cv2.cvtColor(seg, cv2.COLOR_LAB2BGR)
cv2.imwrite("kmeans_seg.jpg", seg_bgr)
Mean-shift treats pixels (colour + optionally position) as samples from an unknown density and climbs the density gradient to a mode using a kernel window:
\[ m(x) = \frac{\sum_i K(x_i - x)\, x_i}{\sum_i K(x_i - x)} - x \]
Unlike k-means, you don’t choose \(K\) — the number of modes falls out of the bandwidth \(h\). OpenCV: cv2.pyrMeanShiftFiltering(img, sp, sr) (spatial and colour radius).
SLIC (Simple Linear Iterative Clustering) is a local k-means in a 5-D space \([L, a, b, x, y]\), restricted to a search window around each cluster — this makes it near-linear time.
Superpixels are a pre-processing step, not a final segmentation — they reduce an image with millions of pixels to a few hundred/thousand perceptually-meaningful atoms for the next stage (graph cuts, classification, etc.).
import cv2
img = cv2.imread("scene.jpg")
slic = cv2.ximgproc.createSuperpixelSLIC(img, algorithm=cv2.ximgproc.SLICO,
region_size=30)
slic.iterate(10)
mask = slic.getLabelContourMask()
overlay = img.copy()
overlay[mask == 255] = (0, 0, 255)
print("Superpixels:", slic.getNumberOfSuperpixels())
cv2.imwrite("slic_overlay.jpg", overlay)
Requires opencv-contrib-python (the ximgproc module).
Given predicted mask \(P\) and ground-truth mask \(G\):
\[ \text{IoU} = \frac{|P \cap G|}{|P \cup G|}, \qquad \text{Dice} = \frac{2|P \cap G|}{|P| + |G|} \]
Treat the (gradient) image as a topographic surface: bright ridges are gradient maxima (object boundaries), dark basins are homogeneous regions. “Flooding” from markers upward fills basins until they meet at ridge lines — those lines are the segmentation boundary.
cv2.watershed(img, markers) — boundaries are written back as \(-1\) in markers.Failure mode: over-segmentation from noisy gradients — always smooth first and seed with good markers (distance transform + connected components is the standard recipe for touching objects, e.g. coins or cells).
Model the image as a graph: one node per pixel plus two terminals source (foreground) and sink (background). Minimise an energy:
\[ E(L) = \sum_p D_p(L_p) + \lambda \sum_{(p,q) \in \mathcal{N}} V_{p,q}(L_p, L_q) \]
GrabCut wraps graph cuts with iterative colour-model refinement, seeded from just a bounding box:
import cv2, numpy as np
img = cv2.imread("object.jpg")
mask = np.zeros(img.shape[:2], np.uint8)
bgd, fgd = np.zeros((1, 65), np.float64), np.zeros((1, 65), np.float64)
rect = (50, 50, img.shape[1] - 100, img.shape[0] - 100)
cv2.grabCut(img, mask, rect, bgd, fgd, 5, cv2.GC_INIT_WITH_RECT)
final = np.where((mask == 2) | (mask == 0), 0, 1).astype("uint8")
out = img * final[:, :, None]
A snake is a parametric curve \(v(s) = (x(s), y(s))\) that deforms to minimise an energy balancing smoothness and image attraction:
\[ E_{\text{snake}} = \int_0^1 \underbrace{\alpha |v'(s)|^2 + \beta |v''(s)|^2}_{\text{internal (smoothness)}} + \underbrace{E_{\text{ext}}(v(s))}_{\text{image energy, e.g. } -|\nabla I|^2} \, ds \]
cv2.snake is legacy/removed in places — skimage.segmentation.active_contour is the practical implementation today.Every classical method above implicitly encodes one or more Gestalt grouping cues that humans use too:
Every method today hand-designs the similarity or energy function. The deep-learning alternative learns a pixel-to-label mapping end-to-end from data:
Classical segmentation doesn’t disappear: GrabCut/SLIC are still used for dataset annotation tools, and IoU/Dice remain the evaluation metrics for the learned models.
| Method | Needs a seed? | Speed | Best for |
|---|---|---|---|
| Region growing | Yes (seed pixel) | Fast | Homogeneous blobs, known start point |
| K-means | No (choose K) | Fast | Colour-distinct regions, quick prototyping |
| Mean-shift | No (choose bandwidth) | Slow | Unknown region count, smooth colour fields |
| SLIC | No | Very fast | Pre-processing / over-segmentation for later stages |
| Watershed | Yes (markers) | Fast | Touching/overlapping convex objects |
| GrabCut | Yes (box) | Moderate | Interactive foreground extraction |
Lab 9 — Build a Real-World CV System.
Deliverable: an end-to-end measuring / defect system logging to a database.
Also due this week: the project proposal for your capstone.
Intelligent Vision Systems; University of Lahore | Dr. Muhammad Kazim