Object Detection — AI Tutorials — Subhendu Datta Bhowmik

Detection Fundamentals

Given an image, produce a set of bounding boxes $\{(x_1, y_1, x_2, y_2, c, s)\}$ where $(x_1, y_1, x_2, y_2)$ are box coordinates, $c$ is the class label, and $s$ is the confidence score.

Intersection over Union (IoU)

$\text{IoU} = \frac{|B_{pred} \cap B_{gt}|}{|B_{pred} \cup B_{gt}|}$

IoU threshold	Typical use
0.5	PASCAL VOC metric
0.75	Strict matching
0.5:0.95	COCO metric (averaged)

Non-Maximum Suppression (NMS)

When many overlapping boxes predict the same object, NMS retains only the highest-scoring box:

Sort boxes by confidence score (descending)
Take top-scoring box; suppress all boxes with IoU > threshold
Repeat until no boxes remain

Mean Average Precision (mAP)

For each class, compute Average Precision (AP) as the area under the Precision-Recall curve. mAP averages AP across all classes:

$\text{mAP} = \frac{1}{|C|} \sum_{c \in C} \text{AP}_c$

COCO uses mAP@[0.5:0.05:0.95] averaged over 10 IoU thresholds.

Architecture Comparison

Property	Faster R-CNN	YOLOv8	DETR
Speed (FPS, GPU)	~15	~160	~28
mAP@COCO	37.4	50.2 (L)	42.0
Anchors	Yes	No	No
NMS required	Yes	Yes	No
Small objects	Good	Moderate	Moderate

IoU and NMS from scratchpython

import torch

def box_iou(boxes_a: torch.Tensor, boxes_b: torch.Tensor) -> torch.Tensor:
    """Pairwise IoU between two sets of boxes in xyxy format."""
    inter_x1 = torch.max(boxes_a[:, None, 0], boxes_b[None, :, 0])
    inter_y1 = torch.max(boxes_a[:, None, 1], boxes_b[None, :, 1])
    inter_x2 = torch.min(boxes_a[:, None, 2], boxes_b[None, :, 2])
    inter_y2 = torch.min(boxes_a[:, None, 3], boxes_b[None, :, 3])
    inter_area = (inter_x2 - inter_x1).clamp(min=0) * (inter_y2 - inter_y1).clamp(min=0)
    area_a = (boxes_a[:, 2] - boxes_a[:, 0]) * (boxes_a[:, 3] - boxes_a[:, 1])
    area_b = (boxes_b[:, 2] - boxes_b[:, 0]) * (boxes_b[:, 3] - boxes_b[:, 1])
    union = area_a[:, None] + area_b[None, :] - inter_area
    return inter_area / union.clamp(min=1e-6)

def nms(boxes: torch.Tensor, scores: torch.Tensor, iou_threshold=0.5):
    order = scores.argsort(descending=True)
    keep = []
    while order.numel() > 0:
        i = order[0].item()
        keep.append(i)
        if order.numel() == 1: break
        ious = box_iou(boxes[i:i+1], boxes[order[1:]])[0]
        order = order[1:][ious <= iou_threshold]
    return torch.tensor(keep, dtype=torch.long)

# Demo
boxes  = torch.tensor([[10,10,50,50],[12,12,52,52],[100,100,150,150]], dtype=torch.float)
scores = torch.tensor([0.9, 0.75, 0.85])
kept   = nms(boxes, scores, iou_threshold=0.5)
print("Kept indices:", kept.tolist())  # [0, 2]

Faster R-CNN: Two-Stage Detection

Stage 1 — Region Proposal Network (RPN):

Slides a small network over the feature map
At each location scores $k$ anchor boxes for "objectness"
Predicts offsets $(\Delta x, \Delta y, \Delta w, \Delta h)$ to refine each anchor
Top-N proposals (typically 300) pass to Stage 2

Stage 2 — RoI Head:

RoI Align extracts fixed-size feature crops using bilinear interpolation (vs integer-aligned RoI Pool)
A small FC network classifies each crop and refines the box

DETR: Detection Transformer

DETR (Carion et al., 2020) frames detection as a set prediction problem:

CNN backbone extracts feature map $f$
Transformer encoder processes flattened features
$N$ learned object queries attend to encoder output via cross-attention
Decoder outputs $N$ box+class predictions in parallel
Hungarian matching assigns predictions to ground-truth boxes

$\hat{\sigma} = \underset{\sigma}{\arg\min} \sum_i \mathcal{L}_{match}(y_i, \hat{y}_{\sigma(i)})$

No NMS needed — each query predicts exactly one unique object.

Faster R-CNN and YOLOv8 for custom detectionpython

import torch
import torchvision
from torchvision.models.detection import fasterrcnn_resnet50_fpn, FasterRCNN_ResNet50_FPN_Weights
from torchvision.models.detection.faster_rcnn import FastRCNNPredictor

# Replace head for custom classes (background + N classes)
def get_fasterrcnn(num_classes: int):
    model = fasterrcnn_resnet50_fpn(weights=FasterRCNN_ResNet50_FPN_Weights.COCO_V1)
    in_features = model.roi_heads.box_predictor.cls_score.in_features
    model.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes)
    return model

model = get_fasterrcnn(num_classes=4)   # background + 3 classes

# Training loop
def train_detection(model, train_loader, epochs=10, lr=5e-4, device='cuda'):
    model = model.to(device)
    optimizer = torch.optim.SGD(
        [p for p in model.parameters() if p.requires_grad],
        lr=lr, momentum=0.9, weight_decay=5e-4
    )
    for epoch in range(epochs):
        model.train()
        for images, targets in train_loader:
            images  = [img.to(device) for img in images]
            targets = [{k: v.to(device) for k, v in t.items()} for t in targets]
            loss_dict = model(images, targets)
            loss = sum(loss_dict.values())
            optimizer.zero_grad(); loss.backward(); optimizer.step()
        print(f"Epoch {epoch+1}  loss={loss.item():.4f}")

# YOLOv8 fine-tuning (pip install ultralytics)
# from ultralytics import YOLO
# model = YOLO('yolov8s.pt')
# model.train(data='dataset.yaml', epochs=100, imgsz=640, batch=16)
# model.export(format='onnx')

Knowledge check

A predicted box has IoU = 0.4 with the ground-truth box. Using PASCAL VOC metric (IoU threshold = 0.5), this detection is:

Knowledge check

What is the key innovation that allows DETR to eliminate Non-Maximum Suppression?

Knowledge check

YOLOv8 switched from anchor-based to anchor-free detection. What is the primary benefit?

← Computer Vision