Camera-based Robotic Sensing and Machine Vision — Robotics Tutorials

Camera Types for Robotic Vision

Cameras are the richest exteroceptive sensors available to robots, providing dense 2D or 3D information about the scene at every frame. Different camera types offer different trade-offs between depth information, resolution, frame rate, and cost.

Monocular cameras capture a standard 2D color or grayscale image using a single lens. They provide no direct depth information — depth must be inferred through motion (Structure from Motion), known object size, or machine learning. They are the lowest cost and simplest to use.

Stereo cameras pair two cameras separated by a known baseline distance. By finding corresponding pixels in the left and right images (stereo matching / disparity estimation) and applying triangulation, dense depth maps are computed. Depth accuracy decreases quadratically with distance; near-range performance is excellent. Examples: ZED (Stereolabs), Intel RealSense 400 series (stereo), Carnegie Robotics MultiSense.

RGB-D cameras combine a color camera with a depth sensor, producing aligned color + depth images. Two depth sensing technologies are used:

Structured light: Projects a known IR pattern onto the scene; deformation of the pattern reveals depth (Intel RealSense D415/D435, Microsoft Kinect v1, Orbbec Astra). Works poorly in outdoor sunlight (overwhelms the IR pattern).
Time-of-Flight (ToF): A modulated IR signal is emitted and the phase shift or pulse delay of the return measures depth (Microsoft Azure Kinect, PMD pico flexx). More robust outdoors but lower resolution than structured light.

Event cameras (Dynamic Vision Sensors, DVS) output asynchronous events for each pixel that detects a brightness change above threshold, rather than synchronous frames. This provides microsecond-resolution motion detection, high dynamic range, and no motion blur — ideal for fast robot motion. Examples: DAVIS346 (IniVation), Prophesee EVK4.

Camera Type Comparison for Robotics

Type	Depth Info	Typical Resolution	Frame Rate	Cost	Primary Use Case
Monocular	None (indirect)	1 – 20 MP	30 – 200 fps	$20 – $500	Object detection, classification, visual odometry
Stereo	Triangulation (0.1 – 10 m)	1 – 4 MP per eye	15 – 60 fps	$100 – $5,000	Mobile robot navigation, 3D reconstruction
RGB-D (structured light)	Structured light (0.1 – 5 m)	640×480 depth + 1080p color	15 – 90 fps	$100 – $800	Manipulation, indoors SLAM, human pose
RGB-D (ToF)	Phase shift ToF (0.3 – 10 m)	512×512 depth + 1080p color	15 – 30 fps	$200 – $1,500	Outdoor RGB-D, body tracking
Event camera	None (indirect via stereo event)	240×180 – 1280×720 events	1 MHz event rate	$200 – $5,000	High-speed motion, high-dynamic-range scenes
360° fisheye	None (indirect)	4 – 12 MP	30 – 60 fps	$100 – $2,000	Omnidirectional navigation, panoramic SLAM

Pinhole Camera Model

The pinhole camera model relates a 3D world point to a 2D image point:

u = f_x × (X / Z) + c_x
v = f_y × (Y / Z) + c_y

In homogeneous form:  λ × [u, v, 1]ᵀ = K × [X, Y, Z]ᵀ

Intrinsic matrix K = | f_x   0   c_x |
                     |  0   f_y  c_y |
                     |  0    0    1  |

where:
  (X, Y, Z) = 3D point in camera frame
  (u, v) = 2D pixel coordinates
  f_x, f_y = focal lengths in pixels (horizontal, vertical)
  c_x, c_y = principal point (image center in pixels)
  λ = projective scale factor (= Z)

For stereo triangulation, disparity d = u_left − u_right relates to depth:
  Z = f_x × baseline / d

Real lenses deviate from the pinhole model due to lens distortion (radial and tangential). Distortion coefficients (k₁, k₂, p₁, p₂, k₃) are determined during calibration and applied to undistort images before processing.

Classical Computer Vision Pipeline

The classical machine vision pipeline processes camera images through a sequence of deterministic steps to extract actionable information:

1. Image acquisition and preprocessing: Debayering (for Bayer-pattern sensors), noise removal (Gaussian blur, median filter), histogram equalization for contrast enhancement, and color space conversion (BGR → grayscale, HSV, or Lab).

2. Edge detection: The Canny edge detector applies Gaussian smoothing → gradient magnitude estimation (Sobel operators: ∂I/∂x, ∂I/∂y) → non-maximum suppression → hysteresis thresholding. Produces thin, connected edge maps robust to noise.

3. Feature extraction:

SIFT (Scale-Invariant Feature Transform): Detects keypoints at multiple scales using Difference-of-Gaussian; computes 128-dimensional gradient orientation histograms — invariant to scale, rotation, and illumination change
ORB (Oriented FAST and Rotated BRIEF): A binary feature descriptor combining FAST keypoint detector and BRIEF descriptor with orientation; 100× faster than SIFT for real-time use (ORB-SLAM2/3 relies on ORB)
Harris corner detector: Detects image corners by analyzing the second-moment matrix of image gradients; computationally simple, widely used

4. Feature matching and geometric reasoning: FLANN (Fast Library for Approximate Nearest Neighbors) matches feature descriptors. RANSAC (Random Sample Consensus) robustly estimates geometric transformations (homography, essential matrix) from noisy correspondences.

5. Object detection (classical): Template matching via normalized cross-correlation; Viola-Jones (Haar cascades) for face detection; HOG + SVM (Histogram of Oriented Gradients + Support Vector Machine) for pedestrian detection.

Deep Learning for Robot Vision

Convolutional Neural Networks (CNNs) have largely superseded classical feature engineering for high-level visual recognition tasks. CNNs learn hierarchical feature representations directly from labeled training data.

Image classification: CNNs (VGG, ResNet, EfficientNet) output a probability distribution over semantic categories. ResNet-50 achieves ~76% top-1 accuracy on ImageNet with 25M parameters, running at ~50 fps on a modern GPU.

Object detection: Two-stage detectors (Faster R-CNN) first propose regions of interest, then classify them — high accuracy but slower (~10 fps). Single-stage detectors (YOLO, SSD, DETR) predict bounding boxes and classes in a single forward pass — YOLOv8 achieves ~50 ms/frame on an NVIDIA Jetson Orin.

Semantic segmentation: Models (DeepLab, SegFormer, Mask2Former) assign a class label to every pixel, enabling robots to understand drivable surfaces, object boundaries, and scene layout at pixel level.

6-DoF object pose estimation: Networks (DOPE by NVIDIA, FoundPose, MegaPose) predict the full 6D pose (3D position + 3D orientation) of known objects for robotic grasping. They typically output 2D keypoint predictions on object CAD model vertices, then use PnP (Perspective-n-Point) to recover the 3D pose.

Visual foundation models: Large pretrained vision-language models (CLIP, SAM — Segment Anything Model, DINOv2) provide zero-shot or few-shot generalization, enabling robots to identify objects from text descriptions without task-specific training.

Visual Servoing: Camera Feedback for Robot Control

Visual servoing uses camera measurements directly in the robot control loop — the camera becomes the primary feedback sensor rather than joint encoders alone. This enables robot motion to be controlled relative to visual features in the image, robust to kinematic model errors and positioning uncertainty.

Image-Based Visual Servoing (IBVS): The control error e is defined in 2D image space: e = s − s*, where s are current image feature coordinates (e.g., point projections, line coordinates) and s* are desired feature coordinates. The control law is: q̇ = −λ × J⁺_image × e, where J_image is the interaction matrix (image Jacobian) mapping robot joint velocities to image feature motion. IBVS does not require 3D pose estimation — control runs entirely in image space.

Position-Based Visual Servoing (PBVS): Camera measurements are used to estimate the 3D pose of the target relative to the camera (using pose estimation algorithms). Control error is defined in 3D Cartesian space: e = T_current − T_desired. The robot is commanded to reduce this 3D pose error. PBVS requires accurate camera calibration and pose estimation.

Hybrid visual servoing and deep visual servoing (replacing the interaction matrix with a learned mapping) are active research directions addressing the limitations of each classical approach.

Classical Computer Vision vs. Deep Learning for Robotics

Classical Computer Vision

Handcrafted feature descriptors (SIFT, ORB, Canny edges)
Deterministic and interpretable — known failure modes
Works well with little or no training data
Robust to domain shift if task is well-defined geometrically
Computationally efficient — runs on CPU in real time
Struggles with occlusion, illumination variation, deformable objects
Still the backbone of visual odometry (ORB-SLAM3) and stereo matching

Deep Learning Computer Vision

Learned feature representations from large labeled datasets
State-of-the-art performance on recognition and detection benchmarks
Handles illumination, viewpoint, and appearance variation robustly
Requires large labeled training datasets and GPU computation
Black-box behavior — difficult to diagnose failure modes
Prone to distribution shift between training and deployment environments
Foundation models (SAM, CLIP) reduce data requirements for new tasks

Key terms

Pinhole camera model: A simplified model of image formation in which light rays from the scene pass through a single point (the optical center) and project onto the image plane; characterized by the intrinsic matrix K.
Intrinsic matrix (K): A 3×3 matrix encoding the internal camera parameters: focal lengths (f_x, f_y) and principal point (c_x, c_y); determined through camera calibration.
Stereo vision: A depth sensing technique using two cameras with a known baseline separation; depth is computed by triangulating corresponding pixels found via stereo matching.
Depth map: An image where each pixel value represents the distance from the camera to the corresponding surface point in the scene.
RGB-D camera: A camera that captures aligned color (RGB) and depth (D) images simultaneously, using structured light or time-of-flight depth sensing.
Visual servoing: A robot control technique that uses camera measurements directly in the feedback control loop to guide robot motion relative to visual features.
SIFT (Scale-Invariant Feature Transform): A classical computer vision algorithm that detects and describes local image features invariant to scale, rotation, and illumination changes; used for feature matching across images.
CNN (Convolutional Neural Network): A deep learning architecture with convolutional layers that learn hierarchical spatial feature representations from images; the foundation of modern visual recognition systems.
Semantic segmentation: A computer vision task that assigns a class label to every pixel in an image, enabling understanding of scene layout and object boundaries at pixel level.
Optical flow: The apparent motion of brightness patterns in an image sequence caused by relative motion between the camera and scene; used for motion estimation and visual odometry.
Homography: A projective transformation mapping points between two image planes; used to relate images of a planar scene taken from different viewpoints, and in visual servoing for planar targets.
Camera calibration: The process of determining intrinsic camera parameters (focal length, principal point, distortion coefficients) from images of a known calibration target (e.g., checkerboard).

← Sensing & Perception