Computer Vision for Robots

ENG100/400 — Lecture 7

  • From simple vision (thresholds, filters) → advanced perception (point clouds, detectors)
  • Hands‑on apps: filtering, edges, convolution, line following, point clouds, detection, neural nets

Agenda

  • Robot line following with LEDs or a camera (thresholding)
  • Image filtering and convolution
  • Edge detection (Sobel/Canny intuition)
  • Point clouds: segmentation, plane fit
  • Object detection (YOLO) with pre‑canned media
  • Detecting people and an orange ping‑pong ball
  • Neural nets under the hood (FF/Backprop)

Simple vision: LED line following

Idea

  • Sense reflectance above/below a threshold across a few sensors (or pixels across a scanline)
  • Estimate line position from sensor pattern; steer proportionally
  • Filters smooth noise; hysteresis avoids chatter

Open line‑following sim

Example texture used for demos

We’ll simulate reflectance sensors and thresholds to follow a path.

Image filters and thresholding

  • Grayscale, brightness/contrast, blur (box/Gaussian), sharpen
  • Binary thresholding: simple vs. adaptive
  • Morphology (erode/dilate) consolidates blobs

Try filter lab

Scene image for filtering

We’ll use local test images to keep everything offline.

Convolution: sliding kernels

  • Convolution = weighted sum in a neighborhood
  • Examples: blur, sharpen, edge/emboss
  • Kernels: 3×3 or 5×5 are common on MCUs

Open kernel playground

Portrait for convolution demo

Edge detection (Sobel/Canny intuition)

  • Compute gradients Gx, Gy (Sobel); magnitude √(Gx²+Gy²)
  • Non‑max suppression and hysteresis (Canny)
  • Edges feed into lane/line and feature tracking

Try edge detector

Edge test image

Point clouds: basic algorithms

  • From depth → 3D points; transform to world frame
  • Segmentation: plane fit (RANSAC), clustering (DBSCAN)
  • Applications: ground removal, obstacle extraction

Open point‑cloud demo

Live preview sparkles for effect (full app is separate).

Object detection (YOLO family)

  • One‑stage detectors: grid → boxes + classes
  • Modern variants: YOLOv5/8/n variants (speed/size)
  • On‑device: use smaller models; or do server‑side

Open YOLO showcase

Pre‑canned YOLO sample image

Using pre‑canned images locally (no heavy model in‑browser).

Detecting people and an orange ping‑pong ball

  • People: detector (YOLO) or HOG+SVM (classic)
  • Orange ball: simple HSV color threshold + blob detect
  • Use morphology to clean the mask; track centroid

Try orange‑ball HSV demo

Scene image

Neural nets: under the hood

  • Layers: linear → nonlinearity (ReLU/σ/tanh)
  • Forward pass: compute activations
  • Backward pass: gradients via chain rule; update weights

Interactive NN explainer

ReLU \( f(x) = \max(0, x) \)

Sigmoid \( \sigma(x) = \frac{1}{1+e^{-x}} \)

tanh \( \tanh(x) = \frac{e^{x}-e^{-x}}{e^{x}+e^{-x}} \)

Derivatives: \( f'(x)=\begin{cases}0 & x<0 \\ 1 & x>0\end{cases} \), \( \sigma'(x)=\sigma(x)(1-\sigma(x)) \), \( (\tanh x)' = 1 - \tanh^2 x \).

We’ll visualize a tiny 2→3→1 net and step gradient descent by hand.

Resources

  • OpenCV docs: https://docs.opencv.org/
  • Ultralytics YOLO: https://docs.ultralytics.com/
  • Point cloud basics: PCL https://pointclouds.org/ (C++)

Media used on these slides are stored in lec7/images for offline viewing.

Further tutorials and references (by topic)

Line following & thresholding

Filters & convolution

Edges

Point clouds

Object detection (YOLO)

Color/HSV

Neural nets & backprop

External links require internet access; the lecture apps and media remain available offline in this repository.