Introduction to Deep Learning and Applications in Robotics — Robotics Tutorials

Convolutional Neural Networks for Robotic Perception

Convolutional Neural Networks (CNNs) are the backbone of robotic perception. Unlike fully connected networks, CNNs exploit the spatial structure of images through convolutional layers (learnable filters that detect local features), pooling layers (spatial downsampling for translation invariance), and fully connected layers (combining spatial features for classification or regression).

Key CNN architectures for robotics:

ResNet (He et al., 2016): residual (skip) connections solve the vanishing gradient problem, enabling networks of 50–200 layers. ResNet-50 is a standard backbone for object detection and semantic segmentation.
EfficientNet: scales width, depth, and resolution together via compound scaling; achieves best accuracy-per-FLOP, important for edge deployment on Jetson or Coral TPU.
MobileNet: depthwise separable convolutions drastically reduce computation (8–9× vs standard convolutions) with modest accuracy loss — suited for real-time inference on embedded hardware.

Object detection architectures:

YOLO (You Only Look Once): single-pass detector predicting bounding boxes and class probabilities on a grid. YOLOv8/v9 achieves 50+ FPS on GPU at 640×640 resolution — the default choice for real-time robot perception.
Faster R-CNN: two-stage detector (Region Proposal Network → RoI classification) with higher accuracy but lower speed; used when precision matters more than latency.
DETR (Detection Transformer): transformer-based end-to-end detector without hand-crafted anchor boxes or NMS post-processing; achieves competitive accuracy with cleaner architecture.

Deep Learning Tasks in Robotics

Task	Architecture	Input	Output	Robot Application
Classification	ResNet, EfficientNet	RGB image (H×W×3)	Class probabilities (softmax)	Object type recognition, terrain classification
Object Detection	YOLOv8, Faster R-CNN, DETR	RGB image	Bounding boxes + class + confidence	Detecting obstacles, tools, products on conveyor
Semantic Segmentation	DeepLabV3+, SegFormer	RGB image	Per-pixel class label	Drivable surface detection, surgical scene parsing
Depth Estimation	DPT, AdaBins, Depth Anything	Mono RGB image	Per-pixel depth map	Monocular 3D perception for single-camera robots
6-DoF Pose Estimation	PVNet, FoundPose, GDR-Net	RGB-D image + object model	R (3×3), t (3×1)	Bin picking, assembly, surgical instrument tracking
Optical Flow	RAFT, FlowFormer	Two sequential frames	Per-pixel displacement vectors	Visual odometry, moving object detection

Reinforcement Learning for Robotics

Reinforcement Learning (RL) trains a robot agent to maximise cumulative reward through interaction with an environment. At each step, the agent observes state s, selects action a according to its policy π(a|s), receives reward r, and transitions to state s'. The goal is to find the optimal policy π* that maximises expected discounted return.

Model-free RL (no dynamics model) is dominant in robot learning:

PPO (Proximal Policy Optimization): on-policy actor-critic algorithm with clipped surrogate objective preventing destructively large policy updates. Widely used for locomotion (MuJoCo Ant, Humanoid) and manipulation. Stable, sample-efficient on parallelised simulators.
SAC (Soft Actor-Critic): off-policy maximum-entropy RL for continuous control. Learns stochastic policies (explores inherently), uses replay buffers for sample efficiency. State-of-the-art on robotic manipulation benchmarks.
TD3 (Twin Delayed DDPG): deterministic off-policy algorithm; twin Q-networks reduce overestimation bias; delayed policy updates for stability.

Sim-to-real pipeline for robot RL:

Design reward function (sparse: success/failure, or dense: shaped reward signal)
Train in parallel simulation (IsaacGym: 4096 environments simultaneously on one GPU)
Apply domain randomisation on physics and sensor parameters
Deploy on hardware with possible fine-tuning (sim-to-real gap correction)

Policy Gradient and Bellman Equation

Core RL optimisation objectives

REINFORCE gradient:  ∇J(θ) = E[∇ log π_θ(a|s) · R]
Bellman equation:    Q(s,a) = r + γ · max_{a'} Q(s',a')
PPO objective:       L_CLIP = E[min(r_t(θ)·Â_t, clip(r_t(θ), 1-ε, 1+ε)·Â_t)]

θ: policy parameters; π_θ(a|s): probability of action a in state s under policy θ; R: cumulative return; γ: discount factor (0 < γ < 1); r_t(θ) = π_θ(a_t|s_t)/π_θ_old(a_t|s_t): probability ratio; Â_t: advantage estimate; ε: PPO clip range (typically 0.2)

The Bellman equation defines the recursive relationship that all Q-learning algorithms solve iteratively. The PPO clipping prevents the new policy from deviating too far from the old policy in a single update step, a key stability mechanism.

Foundation Models in Robotics

Foundation models — large neural networks pre-trained on internet-scale data — are transforming what is possible in robotic manipulation and reasoning.

RT-2 (Robotics Transformer 2, Google DeepMind 2023): A vision-language-action (VLA) model fine-tuned from a large vision-language model (PaLM-E / ViT). The model takes a camera image and a language instruction ("pick up the green cup") and outputs robot joint angles directly as token sequences, without explicit perception or planning modules. RT-2 generalises to novel objects and instructions not seen in robot training data, leveraging the vast semantic knowledge of the base language model.

Octo (Berkeley 2024): Open-source transformer-based robot foundation model trained on the Open X-Embodiment dataset (~800k robot trajectories across 22 robot types). Provides a strong pre-trained backbone that can be fine-tuned to new robots and tasks with relatively few demonstrations.

OpenVLA (Stanford 2024): A 7B-parameter open vision-language-action model based on Llama 2 and DINOv2, achieving state-of-the-art performance on the BridgeV2 dataset. Released with weights and training code.

Implications for robot programming: Rather than writing explicit perception-planning-control code, the emerging paradigm is: (1) collect robot demonstrations, (2) fine-tune a foundation VLA model, (3) deploy. This requires new ROS integration patterns where the VLA model runs as a node consuming image + language goal and publishing joint trajectories.

Supervised Learning vs Reinforcement Learning for Robot Control

Supervised Learning (Imitation Learning)

Data requirement: needs a dataset of expert demonstrations (state, action) pairs
No reward design needed — directly learns to mimic expert behaviour
Sample efficient — learns from hundreds to thousands of demonstrations
Distribution shift: fails on states not covered by demonstrations
Safe deployment: behaviour bounded by the demonstrated expert policy
Best for: manipulation tasks with available human demonstrations, language-conditioned policies
Examples: Behaviour Cloning, DAgger, diffusion policy, RT-2

Reinforcement Learning

Data requirement: only needs environment interactions and a reward signal
Reward design is difficult — sparse rewards cause slow learning, dense rewards may be gamed
Sample inefficient — requires millions to billions of environment steps
Can discover superhuman strategies not present in any demonstration dataset
Deployment safety concerns: RL policies may exploit reward loopholes unsafely
Best for: game-like tasks where simulation is available and reward is clear
Examples: AlphaGo, OpenAI Dactyl, quadruped locomotion (ETH ANYmal), PPO/SAC manipulation

Integrating Deep Learning with ROS 2

Pattern: Deep learning inference as a ROS 2 node

import rclpy
from rclpy.node import Node
from sensor_msgs.msg import Image
from vision_msgs.msg import Detection2DArray
import torch
from cv_bridge import CvBridge  # converts sensor_msgs/Image ↔ numpy array

class YOLODetectorNode(Node):
    def __init__(self):
        super().__init__('yolo_detector')
        self.model = torch.hub.load('ultralytics/yolov8', 'yolov8n').cuda()
        self.bridge = CvBridge()
        self.sub = self.create_subscription(
            Image, '/camera/image_raw', self.image_cb, 10)
        self.pub = self.create_publisher(
            Detection2DArray, '/detections', 10)

    def image_cb(self, msg):
        cv_img = self.bridge.imgmsg_to_cv2(msg, 'bgr8')
        results = self.model(cv_img)
        # convert results to Detection2DArray and publish
        ...

Practical considerations:

Use ros2_numpy or cv_bridge for efficient conversion between ROS messages and numpy/tensor arrays — avoid manual byte-level parsing
NVIDIA Isaac ROS packages provide GPU-accelerated perception components: Isaac ROS NITROS (zero-copy GPU tensor transport), Isaac ROS Object Detection (TAO-optimised YOLO), Isaac ROS Visual SLAM (cuVSLAM)
Use CUDA streams in inference to overlap data transfer and computation, reducing effective latency
Publish at the lowest rate the downstream system requires — most planners do not need >10 Hz detection updates

Key terms

CNN: Convolutional Neural Network — a deep learning architecture using convolutional filters to extract hierarchical spatial features from images, foundational for robotic vision
YOLO: You Only Look Once — a real-time object detection architecture that predicts bounding boxes and class probabilities in a single forward pass, widely deployed in robot perception
reinforcement learning: A machine learning paradigm where an agent learns to maximise cumulative reward through trial-and-error interactions with an environment, without explicit supervision
PPO: Proximal Policy Optimization — an on-policy RL algorithm using a clipped surrogate objective to prevent large policy updates, widely used for robot locomotion training
SAC: Soft Actor-Critic — an off-policy maximum-entropy RL algorithm for continuous control that learns inherently stochastic policies via entropy regularisation; state-of-the-art for manipulation
policy gradient: A class of RL algorithms that directly optimise the policy parameters by computing and following the gradient of expected return with respect to policy parameters
Bellman equation: The recursive optimality condition for Q-values: Q(s,a) = r + γ·max Q(s',a'), relating the value of a state-action pair to the immediate reward plus discounted future value
sim-to-real: The process of training a robot policy in simulation and deploying it on a physical robot, bridging the reality gap via domain randomisation and adaptation techniques
foundation model: A large neural network pre-trained on internet-scale data that can be fine-tuned for downstream robot tasks; examples include RT-2, Octo, and OpenVLA
RT-2: Robotics Transformer 2 — a vision-language-action foundation model from Google DeepMind that directly outputs robot joint commands from camera images and language instructions
transformer: A neural network architecture based on multi-head self-attention, enabling parallel processing of sequences; now central to object detection (DETR), robot learning (RT-2), and NLP
semantic segmentation: A deep learning task assigning a semantic class label to every pixel in an image; used in robots for identifying drivable surfaces, objects, and obstacles
Isaac ROS: NVIDIA's suite of GPU-accelerated ROS 2 packages for robot perception, including zero-copy tensor transport (NITROS), visual SLAM (cuVSLAM), and optimised object detection

← Modelling in ROS, AI and Machine Learning