Convolutional Neural Networks (CNNs)

Why CNNs for Images?

A fully connected network applied to a 224×224×3 image would need 150,528 input connections per neuron in the first layer — millions of parameters just for one layer. This doesn't scale, and FC networks ignore the spatial structure of images.

CNNs solve this with three key inductive biases:

1. Local connectivity: neurons connect only to a local patch (receptive field), not the entire input
2. Parameter sharing: the same filter (kernel) is applied across all positions → drastically fewer parameters
3. Translation invariance: detecting an edge is the same operation regardless of where it is in the image

The Convolution Operation

For a 2D input $X$ and filter $W$ of size $k \times k$:

$(X * W)[i,j] = \sum_{m=0}^{k-1} \sum_{n=0}^{k-1} X[i+m, j+n] \cdot W[m,n]$

Key Parameters

Parameter	Effect
Kernel size	3×3 (most common), 5×5, 7×7 — larger = larger receptive field
Stride	Step size of the sliding window; stride=2 halves spatial dimensions
Padding	"same" keeps spatial size; "valid" shrinks output
# filters	Determines depth of output feature map

Output Spatial Size

$\text{Output size} = \left\lfloor \frac{\text{Input size} - k + 2p}{s} \right\rfloor + 1$

where $k$ = kernel size, $p$ = padding, $s$ = stride.

What Filters Learn

• Layer 1: edges, colors, gradients
• Layer 2: textures, corners, simple patterns
• Layer 3+: object parts, complex textures
• Deep layers: object-level concepts (faces, wheels, etc.)

Pooling Layers

Pooling downsamples feature maps, reducing spatial dimensions and building invariance to small translations.

Max Pooling

Takes the maximum value in each pooling window: $y[i,j] = \max_{m,n} x[si+m, sj+n]$

• Most common: 2×2 with stride 2 → halves height and width
• Retains the most prominent feature activation in each region

Average Pooling

Takes the mean of the window. Less common in hidden layers; used in:

• Global Average Pooling (GAP): averages each feature map to a single value → replaces flatten + FC layers in modern architectures (reduces overfitting, fewer parameters)

No Pooling — Stride Convolutions

Many modern architectures (ResNet v2, EfficientNet) replace max pooling with stride-2 convolutions, keeping all spatial information longer.

CNN Architecture Evolution

LeNet-5 (1998)

Conv → Pool → Conv → Pool → FC → FC. First successful CNN for digit recognition (MNIST).

AlexNet (2012) — The ImageNet Moment

5 conv layers + 3 FC layers. Introduced: ReLU activations, dropout, GPU training, data augmentation. Won ImageNet with 15.3% top-5 error (previous SOTA: 26%).

VGG (2014)

All 3×3 convolutions, very deep (16–19 layers). Simple, uniform architecture — two 3×3 convs have the same receptive field as one 5×5 but fewer parameters.

ResNet (2015) — Residual Connections

Introduced skip connections (residual blocks) that allow gradients to flow directly:

$\mathbf{y} = F(\mathbf{x}, \{W_i\}) + \mathbf{x}$

Enabled training 50, 101, 152+ layer networks. Still widely used as backbone.

Modern: EfficientNet, ConvNeXt

• EfficientNet: compound scaling (width, depth, resolution simultaneously)
• ConvNeXt: CNN design inspired by Vision Transformers, competitive with ViTs

import torch
import torch.nn as nn
import torchvision
import torchvision.transforms as transforms
from torch.utils.data import DataLoader

# ─── Data ────────────────────────────────────────────────────
transform_train = transforms.Compose([
    transforms.RandomHorizontalFlip(),
    transforms.RandomCrop(32, padding=4),
    transforms.ToTensor(),
    transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)),
])
transform_test = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)),
])

train_set = torchvision.datasets.CIFAR10(root="./data", train=True, download=True, transform=transform_train)
test_set  = torchvision.datasets.CIFAR10(root="./data", train=False, download=True, transform=transform_test)
train_loader = DataLoader(train_set, batch_size=128, shuffle=True, num_workers=2)
test_loader  = DataLoader(test_set,  batch_size=256, shuffle=False, num_workers=2)

# ─── CNN with Residual Block ──────────────────────────────────
class ResBlock(nn.Module):
    def __init__(self, channels):
        super().__init__()
        self.block = nn.Sequential(
            nn.Conv2d(channels, channels, 3, padding=1, bias=False),
            nn.BatchNorm2d(channels), nn.ReLU(inplace=True),
            nn.Conv2d(channels, channels, 3, padding=1, bias=False),
            nn.BatchNorm2d(channels),
        )
        self.relu = nn.ReLU(inplace=True)

    def forward(self, x):
        return self.relu(self.block(x) + x)  # skip connection

class SmallResNet(nn.Module):
    def __init__(self, num_classes=10):
        super().__init__()
        self.stem = nn.Sequential(
            nn.Conv2d(3, 64, 3, padding=1, bias=False),
            nn.BatchNorm2d(64), nn.ReLU(inplace=True),
        )
        self.stage1 = nn.Sequential(ResBlock(64), ResBlock(64))
        self.stage2 = nn.Sequential(
            nn.Conv2d(64, 128, 3, stride=2, padding=1, bias=False),
            nn.BatchNorm2d(128), nn.ReLU(inplace=True),
            ResBlock(128),
        )
        self.stage3 = nn.Sequential(
            nn.Conv2d(128, 256, 3, stride=2, padding=1, bias=False),
            nn.BatchNorm2d(256), nn.ReLU(inplace=True),
            ResBlock(256),
        )
        self.head = nn.Sequential(
            nn.AdaptiveAvgPool2d(1),   # Global Average Pooling
            nn.Flatten(),
            nn.Linear(256, num_classes),
        )

    def forward(self, x):
        return self.head(self.stage3(self.stage2(self.stage1(self.stem(x)))))

model = SmallResNet().cuda()
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3, weight_decay=1e-4)
criterion = nn.CrossEntropyLoss()

# Training loop (abbreviated)
for epoch in range(30):
    model.train()
    for imgs, labels in train_loader:
        imgs, labels = imgs.cuda(), labels.cuda()
        loss = criterion(model(imgs), labels)
        optimizer.zero_grad(); loss.backward(); optimizer.step()

Transfer Learning

Training a CNN from scratch requires millions of labeled images and days of compute. Transfer learning reuses a pretrained backbone:

Approach 1: Feature Extraction

Freeze all pretrained layers; only train the new classification head:

model = torchvision.models.resnet50(weights="IMAGENET1K_V2")
for param in model.parameters():
    param.requires_grad = False
model.fc = nn.Linear(2048, num_classes)  # replace head
# Only model.fc parameters will be updated

Approach 2: Fine-Tuning

Unfreeze all or some layers and train with a small learning rate:

for param in model.parameters():
    param.requires_grad = True
optimizer = torch.optim.Adam([
    {"params": model.layer4.parameters(), "lr": 1e-4},
    {"params": model.fc.parameters(), "lr": 1e-3},
])

Rule of thumb: feature extraction when data is small (<1K) and similar to ImageNet; fine-tune all layers when you have more data or different domain.

Summary

• Convolution slides learnable filters over input, exploiting local connectivity and parameter sharing
• Pooling downsamples feature maps and builds spatial invariance
• Architecture evolution: LeNet → AlexNet → VGG → ResNet → EfficientNet/ConvNeXt
• Residual connections (ResNet) solved the vanishing gradient problem for very deep networks
• Transfer learning lets you use pretrained ImageNet weights for new tasks with minimal data

Next: RNN, LSTM & GRU — sequential models for text, time series, and any ordered data.