Lecture 10 — Calibration & Stereo

Lecture 10

Dr. Muhammad Kazim

Department of Intelligent Systems, University of Lahore

Today’s Roadmap

L19 — Calibration: intrinsic/extrinsic parameters; lens distortion models; Zhang’s planar calibration; reprojection error; undistortion & rectification.
L20 — Stereo: epipolar geometry; fundamental & essential matrices; rectification; block-matching & SGBM disparity; depth from disparity; RGB-D sensors.

Reading: Szeliski 2e §11.1–12.1. Lab 10 this week: calibration, stereo & depth on the Pi.

The Pinhole Camera, Formally

A 3-D point \(X = (X, Y, Z)\) projects to pixel \(x = (u, v)\) through:

\[ s \begin{bmatrix} u \\ v \\ 1 \end{bmatrix} = K \, [R \mid t] \begin{bmatrix} X \\ Y \\ Z \\ 1 \end{bmatrix}, \qquad K = \begin{bmatrix} f_x & 0 & c_x \\ 0 & f_y & c_y \\ 0 & 0 & 1 \end{bmatrix}

\(K\) — intrinsic matrix: focal lengths \((f_x, f_y)\) in pixels, principal point \((c_x, c_y)\).
\([R \mid t]\) — extrinsic matrix: the camera’s rotation and translation relative to the world frame.
\(s\) — the (unknown) depth/scale factor, recovered from \(Z\) after dividing through.

Why Calibrate?

Every consumer/embedded lens introduces geometric distortion — straight lines in the world bow in the image.
Metric tasks — measuring object size, stereo depth, AR overlay, SLAM — are meaningless without knowing \(K\) and the distortion coefficients.
Calibration is a one-time-per-camera-setup step: it must be redone if the lens, focus, or sensor changes.

Lens Distortion Models

Radial distortion (barrel/pincushion), the dominant term for most lenses:

\[ x_{\text{distorted}} = x(1 + k_1 r^2 + k_2 r^4 + k_3 r^6) \]

Tangential distortion (lens not perfectly parallel to the sensor):

\[ x_{\text{distorted}} = x + [2 p_1 x y + p_2(r^2 + 2x^2)] \]

OpenCV’s standard distortion vector is \((k_1, k_2, p_1, p_2, k_3)\). Wide-angle / fisheye lenses (common on embedded cameras) need the separate fisheye model — cv2.fisheye.*.

Zhang’s Planar Calibration Method

Zhang (2000) recovers \(K\) and distortion from ≥ 2 views of a flat pattern (checkerboard) at different unknown poses — no special calibration rig needed.

Capture 15–25 images of a checkerboard from varied angles/distances, covering the frame corners as well as the centre.
Detect chessboard corners in each image (cv2.findChessboardCorners), refine to sub-pixel accuracy.
Each view gives a homography between the known planar board and the image; combine homographies across views to solve for \(K\) (closed form), then refine \(K\), distortion, and every view’s \([R\mid t]\) jointly by non-linear least squares (Levenberg–Marquardt).

Reprojection Error

The calibration’s optimisation objective — and its quality metric — is the reprojection error:

\[ E = \frac{1}{N}\sum_{i=1}^{N} \big\lVert x_i - \hat{x}_i(K, R, t, k) \big\rVert^2 \]

where \(x_i\) are the detected corners and \(\hat{x}_i\) is where the calibrated model predicts they should project.

Target: RMS reprojection error < 0.5 px. Above ~1 px, suspect a bad board size, blurry images, or too few varied poses.
Always calibrate at the resolution/focus you will actually run at — intrinsics change with both.

Code — Chessboard Calibration

import cv2, numpy as np, glob

cols, rows = 9, 6                       # inner corners
objp = np.zeros((rows * cols, 3), np.float32)
objp[:, :2] = np.mgrid[0:cols, 0:rows].T.reshape(-1, 2)

objpoints, imgpoints = [], []
for f in glob.glob("calib/*.jpg"):
    gray = cv2.imread(f, cv2.IMREAD_GRAYSCALE)
    ok, corners = cv2.findChessboardCorners(gray, (cols, rows))
    if ok:
        corners = cv2.cornerSubPix(gray, corners, (11, 11), (-1, -1),
            (cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 30, 0.001))
        objpoints.append(objp); imgpoints.append(corners)

rms, K, dist, rvecs, tvecs = cv2.calibrateCamera(
    objpoints, imgpoints, gray.shape[::-1], None, None)
print(f"RMS reprojection error: {rms:.3f} px")
np.savez("camera_intrinsics.npz", K=K, dist=dist)

Undistortion & Rectification

data = np.load("camera_intrinsics.npz")
K, dist = data["K"], data["dist"]

frame = cv2.imread("test.jpg")
h, w = frame.shape[:2]
newK, roi = cv2.getOptimalNewCameraMatrix(K, dist, (w, h), alpha=0)
undist = cv2.undistort(frame, K, dist, None, newK)
x, y, w2, h2 = roi
undist = undist[y:y+h2, x:x+w2]           # crop the black border

Undistortion removes lens distortion from a single image. Rectification (stereo) additionally re-projects a pair of images onto a common plane so corresponding points share the same row — the prerequisite for fast disparity search, covered next.

Two-View / Epipolar Geometry

For a point \(X\) seen by two cameras, its projections \(x, x'\) and the two camera centres are coplanar — the epipolar constraint:

\[ x'^{\top} F x = 0 \]

\(F\) — the fundamental matrix (3×3, rank 2): relates pixel coordinates in uncalibrated cameras.
\(E = K'^{\top} F K\) — the essential matrix: relates normalised (calibrated) coordinates; encodes exactly the relative rotation and translation \((R, t)\) between the two cameras (up to scale).
Given a point in the left image, its match in the right image must lie on the corresponding epipolar line — this turns 2-D search into 1-D search.

Stereo Rectification

Rectification warps both images so that epipolar lines become horizontal and aligned row-by-row — then correspondence search is a simple 1-D scan along each row.

Calibrate both cameras individually, then jointly (cv2.stereoCalibrate) to get \((R, t)\) between them.
cv2.stereoRectify computes rectifying rotations and new projection matrices.
cv2.initUndistortRectifyMap + cv2.remap apply the warp to each frame.

After rectification: \(\text{disparity} = u_{\text{left}} - u_{\text{right}}\) for the same row \(v\).

Disparity & Depth

For a rectified stereo pair with baseline \(B\) and focal length \(f\) (pixels):

\[ Z = \frac{f B}{d}, \qquad d = u_{\text{left}} - u_{\text{right}} \]

Depth is inversely proportional to disparity — nearby objects have large disparity (easy, precise); far objects have small disparity (hard, noisy).
Wider baseline → better far-range precision, but a larger “blind” minimum range and more occlusion.

Block-Matching & SGBM Disparity

Block matching (BM): for each pixel, slide a small window along the corresponding epipolar row in the other image and pick the offset with minimum cost (SAD/SSD). Fast, but noisy in textureless regions.
Semi-Global Block Matching (SGBM): adds a smoothness penalty aggregated along several 1-D directions through the disparity image, approximating a full 2-D global optimisation efficiently. Much better edges and textureless-region behaviour, at higher compute cost.

stereo = cv2.StereoSGBM_create(
    minDisparity=0, numDisparities=128, blockSize=5,
    P1=8 * 3 * 5**2, P2=32 * 3 * 5**2,
    disp12MaxDiff=1, uniquenessRatio=10,
    speckleWindowSize=100, speckleRange=32)
disp = stereo.compute(left_rect, right_rect).astype(np.float32) / 16.0
depth = (fx * baseline) / (disp + 1e-6)

RGB-D Sensors

Active alternatives to passive stereo, useful when the scene lacks texture for block matching:

Structured light (e.g. early Kinect) — project a known IR pattern, measure its distortion.
Time-of-Flight (ToF) — measure the phase/time delay of a modulated IR signal per pixel.
Active stereo (e.g. Intel RealSense) — stereo pair + an IR projector to add texture to blank surfaces.

Trade-offs vs passive stereo: works in the dark/on blank walls, but limited outdoor range (IR swamped by sunlight), extra power/cost, and a fixed baseline you can’t choose.

Common Pitfalls / Practitioner Notes

Calibrating with too few or too-similar board poses — vary tilt, distance and in-frame position; include the corners of the field of view.
A blurry or partially-out-of-frame board silently degrades \(K\) — always print the RMS error and re-shoot if > 0.5–1 px.
Forgetting to re-calibrate after changing focus/zoom — intrinsics are focus-dependent on most lenses.
Skipping rectification and running SGBM directly on raw stereo pairs — disparity search assumes row-aligned epipolar lines.
Textureless surfaces (a white wall) starve block matching of correspondences — project texture, use RGB-D, or fall back to monocular depth cues.

This Week’s Lab

Lab 10 — Calibration, Stereo & Depth on the Pi.

Calibrate the Pi camera; save camera_intrinsics.npz; report RMS reprojection error.
Undistort a live/still frame and confirm straight lines are straight.
Stereo-calibrate a pair (or use a supplied rectified pair) and compute a disparity map with SGBM.
Convert disparity to a coloured depth map.

Deliverable: a depth/disparity map on a real scene.

Key Takeaways

The camera model factors into intrinsics \(K\), distortion, and extrinsics \([R\mid t]\) — Zhang’s method recovers all of them from a moving planar checkerboard.
Reprojection error is both the calibration objective and your quality gate (< 0.5 px target).
The fundamental/essential matrices encode epipolar geometry; rectification turns 2-D stereo correspondence into a 1-D search.
Depth from disparity: \(Z = fB/d\) — precise near, noisy far, and undefined at zero disparity.