Lecture 14 — Edge Vision & the OpenCV 5.0 Deep Dive

Lecture 14

Dr. Muhammad Kazim

Department of Intelligent Systems, University of Lahore

Agenda

Part A — Edge vision platforms: accelerators, inference runtimes, memory/latency/power, quantisation, pruning, distillation, operator fusion.
Part B — OpenCV 5.0 deep dive: architecture, breaking changes, the new DNN engine, hardware acceleration, migration.

This lecture is the theory behind Lab 14: migrating an OpenCV 4 pipeline to OpenCV 5.0 and deploying a YOLOv8n ONNX detector on the Raspberry Pi 5 + Hailo.

Why Run Inference at the Edge?

Latency: no network round-trip to a cloud GPU — essential for anything closing a control loop (drone, robot, safety system).
Offline operation: keeps working with no internet — field robotics, factories, remote sites.
Privacy & bandwidth: raw video never leaves the device; only results are logged/transmitted.
Cost at scale: no per-inference cloud bill once the hardware is deployed.
Cost of admission: you must fit the model in a fraction of a desktop GPU's memory and power budget — the rest of this lecture is about paying that cost well.

The Compute-Tier Ladder

Tier	Example	Typical power	When to use
MCU	ESP32-CAM	~mW	always-on, no host, cheapest, tiny models (FOMO)
CPU	Raspberry Pi 5	~W	flexible, fast to prototype, moderate FPS
NPU	Pi 5 + Hailo AI HAT	~W (+ NPU)	real-time, offline, production detection
GPU	desktop / Jetson / Colab	tens–hundreds W	training, or edge only when power is not constrained

Every project in this course should end with a filled-in version of this table with your own measured FPS and power numbers, and a justification for the tier you chose (Lab 14 benchmark, Appendix C of the applied-CV handbook).

Accelerator Types

GPU: thousands of simple cores, excellent at the dense matrix multiplies inside conv layers; power-hungry, best when power is not the constraint.
NPU (Neural Processing Unit): fixed-function silicon specialised for the small set of operations neural nets actually use (conv, matmul, activation) — far better performance-per-watt than a GPU for inference. The Hailo AI HAT on the Pi 5 is this course's NPU.
DSP: optimised for structured signal-processing math (filters, FFTs); some SoCs offload parts of a vision pipeline (e.g. ISP work) to a DSP.
FPGA: reconfigurable logic — lets you implement a custom dataflow, at the cost of a much harder development workflow than GPU/NPU.

Inference Runtimes

Runtime	Primary target	Notes
TensorRT	NVIDIA GPU/Jetson	Aggressive graph/kernel fusion, FP16/INT8
TFLite	Mobile / MCU	Small runtime; TFLite Micro targets MCUs like the ESP32
ONNX Runtime	Cross-platform	Runs the ONNX interchange format across many backends (CPU, GPU, NPU execution providers)
OpenVINO	Intel CPU/GPU/VPU	Intel's optimised inference toolkit
HailoRT	Hailo NPU	Runs a compiled `.hef` on the Hailo AI HAT — this course's main edge deployment path

Practical pattern this course uses: train in PyTorch/Ultralytics → export to ONNX (a common interchange format) → either run directly in ONNX Runtime, or compile further for a specific NPU (Hailo DFC → .hef).

Memory Hierarchy, DMA & Zero-Copy

Every extra copy of a frame (sensor → ISP → CPU buffer → inference buffer → display buffer) costs time and power — and at 640×480×3 bytes per frame, copies add up fast at 30 FPS.
DMA (Direct Memory Access): lets peripherals (camera, NPU) move data to/from RAM without the CPU copying it byte-by-byte.
Zero-copy: design the pipeline so the same buffer is reused across stages (e.g. hand the camera's DMA buffer directly to the inference engine) instead of allocating and copying a fresh array each stage.
In picamera2, requesting a small "lores" stream sized exactly to the model's input (see the Hailo code on the next-but-one slide) avoids an extra CPU-side resize of the full-resolution frame on every loop iteration.

Latency, Throughput & Power

Latency: time from frame capture to result available — what matters for a closed control loop (Lecture 11's tracking, or a drone reacting to a detection).
Throughput: frames processed per second, typically measured on a steady stream — what matters for a dashboard/analytics system that can buffer.
Power: energy per unit time (W); combined with latency/throughput gives energy per inference — the number that matters for battery-powered deployments.
Measurement discipline: always report hardware, input resolution, batch size (usually 1 for edge), and whether you measured end-to-end (capture→draw) or model-only latency — these are not interchangeable and mixing them invalidates a comparison.

Quantisation: PTQ vs. QAT

Neural nets are trained in 32-bit floating point, but inference in INT8 is roughly 4× smaller and often several times faster on hardware with INT8 execution units (including the Hailo NPU).
Post-Training Quantisation (PTQ): quantise an already-trained FP32 model using a small calibration set to estimate each layer's activation range — cheap (no retraining) but can lose accuracy on sensitive layers.
Quantisation-Aware Training (QAT): simulate quantisation noise during training so the network learns weights that are robust to it — better accuracy retention, at the cost of a full retraining run.
Default workflow this course uses: PTQ with a calibration set (the Hailo DFC's optimize step, see two slides ahead) — reach for QAT only if PTQ's accuracy drop is unacceptable.

INT8 / FP16 Numerics

FP16 (half precision): halves memory and often roughly doubles throughput vs FP32 on hardware with FP16 execution units, with minimal accuracy loss — usually a "free" first optimisation.
INT8: represents each value as an 8-bit integer plus a per-tensor (or per-channel) scale factor: \(x \approx s \cdot q\), \(q \in [-128,127]\). Requires calibration (PTQ) or quant-aware training (QAT) to choose good scale factors.
Practitioner note: always benchmark accuracy (mAP/IoU/accuracy) after quantisation, not just speed — a fast model that silently lost 15 mAP points is not a win.

Pruning, Distillation & Operator Fusion

Structured pruning: remove entire channels/filters that contribute least — shrinks the model in a way that's easy for hardware to exploit (fewer channels = a genuinely smaller matmul).
Unstructured pruning: zero out individual low-magnitude weights — high theoretical sparsity, but needs sparse-matrix hardware support to actually speed things up; often not worth it on embedded CPUs/NPUs without that support.
Knowledge distillation: train a small "student" network to match a larger "teacher" network's soft output distribution, not just the hard labels — often recovers accuracy lost by aggressively shrinking a model.
Operator fusion: the compiler merges adjacent ops (e.g. conv + batch-norm + ReLU) into one kernel, cutting memory traffic between ops — this is exactly what the Hailo DFC's optimize step and TensorRT's builder do automatically.

The Deploy Chain: `.pt` → ONNX → Hailo `.hef`

# 1. TRAIN (GPU machine / Colab)
yolo detect train data=data.yaml model=yolov8n.pt epochs=100 imgsz=640

# 2. EXPORT to ONNX (opset 11 — a safe target for the Hailo parser)
yolo export model=best.pt format=onnx opset=11 imgsz=640

# 3. COMPILE in the Hailo Dataflow Compiler (Docker container)
hailo parser onnx best.onnx --hw-arch hailo8l --har-path best.har
hailo optimize best.har --hw-arch hailo8l \
    --calib-set-path datasets/mydata/calib --output-har-path best_optimized.har
hailo compiler best_optimized.har --hw-arch hailo8l --output-dir ./
# -> best.hef, copied to the Raspberry Pi

# 4. DEPLOY on the Pi + Hailo (Lab 14 / 06_hailo_picamera2_detect.py)

Golden rule: keep the training environment and the Hailo DFC (compile) environment in separate venvs/containers — they have conflicting numpy/torch requirements.
--hw-arch must match your exact chip (hailo8 / hailo8l / hailo10h); the optimize step is where INT8 calibration happens.

Code: Running a Compiled Model on the Hailo NPU

from picamera2 import Picamera2
from picamera2.devices import Hailo

with Hailo("best.hef") as hailo:
    model_h, model_w, _ = hailo.get_input_shape()
    picam2 = Picamera2()
    cfg = picam2.create_preview_configuration(
        main={"size": (1280, 720), "format": "RGB888"},
        lores={"size": (model_w, model_h), "format": "RGB888"})  # zero-copy-sized inference stream
    picam2.configure(cfg); picam2.start()

    while True:
        disp = picam2.capture_array("main")
        lores = picam2.capture_array("lores")
        results = hailo.run(lores)              # NPU inference
        dets = extract_detections(results, disp.shape[1], disp.shape[0], labels, conf=0.4)

Full script: applied-cv-course/code/06_hailo_picamera2_detect.py. Always diff the Hailo API against the current official picamera2/hailo-apps examples before a live class — the API evolves with HailoRT versions.

Practitioner Notes: Benchmarking Honestly

Always report a CPU baseline (Lecture 4's FPS-vs-resolution table) alongside every accelerated number — "30 FPS" means nothing without the comparison it beats.
Warm up the model (first inference includes one-time setup cost) before timing; measure over many frames, not one.
Separate model-only latency from end-to-end latency (capture + pre-process + inference + post-process + draw) — Lab 14's benchmark table should report both.
Watch for thermal throttling on sustained workloads — a benchmark run for 10 seconds can look very different from one run for 10 minutes on a passively-cooled Pi.

Part B — The OpenCV 5.0 Deep Dive

OpenCV has underpinned computer vision for two decades. OpenCV 5.0 brings a modernised architecture, a redesigned DNN engine, first-class ONNX support and stronger hardware acceleration — with API changes that affect existing code.

Introduction & Architecture

Why OpenCV 5 was introduced: a large legacy C API surface, an ageing build system, and a DNN module that had outgrown its original design were all overdue for modernisation.
OpenCV 5.0 modernises the core architecture and build system, tightens the C++ API, and removes long-deprecated legacy code paths carried since OpenCV 2.x/3.x.
The opencv-contrib vs core-module split, and which algorithms live where, is also reorganised — always check the current module map when porting code that used a contrib-only function.

Breaking Changes & Modules

Deprecated APIs: some OpenCV 4 constructors/functions kept only for backward compatibility in 4.x are removed outright in 5.0 — a straight upgrade of a large OpenCV 4 codebase will not compile without changes.
Moved/renamed modules: some functionality is reorganised between core, contrib, and the DNN module — an import that worked in 4.x may need a new module path in 5.0.
Cascade classifiers: the classic cv2.CascadeClassifier workflow (Lecture 12) survives, but is affected by 5.0's C API cleanup — Lab 14 has you fix a real CascadeClassifier call broken by the migration.
Practitioner rule: never assume an OpenCV 4 snippet "just works" on 5.0 — check the official 4→5 migration guide (docs.opencv.org) for the exact function you're calling before you debug a mystery failure.

The New DNN Engine

OpenCV 5.0 ships a redesigned cv2.dnn engine with first-class ONNX support as the primary interchange format for modern deep-learning models — the same ONNX file you exported from Ultralytics/PyTorch in Lecture 13/14's deploy chain.
Improved graph optimisation and operator coverage close much of the gap with dedicated runtimes (ONNX Runtime, TensorRT) for common CNN detector/classifier graphs, while keeping the familiar OpenCV C++/Python API.
Typical pattern: net = cv2.dnn.readNetFromONNX("best.onnx"), set a backend/target (CPU or an available accelerator), then net.forward() on a preprocessed blob — the same shape of code as OpenCV 4's DNN module, but faster and with wider ONNX op support.

Data Types & Hardware Acceleration

Broader, more consistent FP16 support throughout the pipeline (not just the DNN module) reduces memory traffic on hardware with FP16 execution units.
Backend/target selection lets the same cv2.dnn code run on CPU, GPU (where available), or a supported AI accelerator without rewriting the inference call — the accelerator target is chosen, not the algorithm.
The result is one OpenCV codebase that can target very different compute tiers (Lecture 14's ladder) by changing a backend/target flag, rather than swapping runtimes entirely.

Migration Strategies

Pin your current OpenCV 4.x version and get your test suite/demo passing before touching anything — you need a known-good baseline to diff against.
Upgrade in an isolated virtual environment, not in place — keep the 4.x environment available for comparison.
Run your code and read every deprecation/removal error as a checklist item, one at a time; consult the official 4→5 migration guide for the replacement API.
Re-run your accuracy/behaviour checks after each fix — a code change that "compiles" is not the same as one that behaves identically.
Re-benchmark: migrating is also your opportunity to pick up the new DNN engine's performance gains — measure them, don't assume them.

Live / Lab Demos

# Migrated OpenCV 5.0 pattern: ONNX detector through the new DNN engine
import cv2
net = cv2.dnn.readNetFromONNX("yolov8n.onnx")
blob = cv2.dnn.blobFromImage(frame, scalefactor=1/255.0, size=(640, 640), swapRB=True)
net.setInput(blob)
outputs = net.forward()
# ... decode boxes/scores/classes from `outputs`, then NMS (Lecture 13) ...

Lab 14 walks through: installing OpenCV 5.0, migrating a small OpenCV 4 pipeline, fixing a broken CascadeClassifier call, and running YOLOv8n ONNX detection through the new DNN engine.

Outcome

By the end of this lecture and Lab 14, you should be able to confidently migrate, optimise and modernise a production computer-vision pipeline from OpenCV 4 to OpenCV 5 — recognising breaking changes before they cause a mystery bug, and using the new DNN engine's ONNX support and hardware-acceleration hooks deliberately rather than by accident.

This Week’s Lab

Lab 14 — Modern Deployment with OpenCV 5.0 & Edge Runtimes.

OpenCV 5.0 on the Pi: migrate an OpenCV 4 pipeline (deprecated APIs, moved modules), fix a CascadeClassifier call, and run YOLOv8n via ONNX in the new DNN engine with hardware acceleration.
Export to TFLite/ONNX, run on ONNX Runtime / Hailo, apply INT8 quantisation, and benchmark latency, throughput and power vs. the CPU baseline.

Deliverable: a migrated OpenCV 5 pipeline + YOLOv8n ONNX detection + a quantised model with a benchmark table.

Key Takeaways

Pick a compute tier (MCU/CPU/NPU/GPU) by measured FPS-per-watt for your model on your hardware, not by assumption.
Quantisation (PTQ by default, QAT when accuracy demands it), pruning, distillation and operator fusion are the standard toolbox for shrinking a model to fit an edge budget — always re-measure accuracy after applying them.
The deploy chain .pt → ONNX → (optionally) hardware-specific compiler → deployed model is the same shape whether the target is Hailo, TFLite, or OpenCV 5's DNN engine.
OpenCV 5.0 is a breaking upgrade with real benefits (new DNN engine, ONNX, hardware acceleration) — migrate deliberately, one deprecation at a time, with a working baseline to diff against.