CNNs in Depth — AI Tutorials — Subhendu Datta Bhowmik

The Convolution Operation

A 2D convolution applies a learnable kernel $K$ of size $k \times k$ across an input feature map:

$\text{Output}[i,j] = \sum_{m=0}^{k-1} \sum_{n=0}^{k-1} \text{Input}[i \cdot s + m,\; j \cdot s + n] \cdot K[m, n]$

where $s$ is the stride.

Output Shape Formula

$H_{out} = \left\lfloor \frac{H_{in} + 2p - k}{s} \right\rfloor + 1$

Same formula applies for width $W$ .

Parameter Count

For a convolutional layer with $C_{in}$ input channels, $C_{out}$ output channels, kernel size $k$ :

$\text{Parameters} = C_{out} \times (C_{in} \times k \times k + 1_{bias})$

Receptive Field

The receptive field of a unit is the region of the input image it "sees". For a stack of $L$ convolutional layers each with kernel size $k$ and stride 1:

$RF_L = 1 + L \times (k - 1)$

With stride $s > 1$ or pooling, the RF grows faster: $RF = RF_{prev} + (k-1) \times \prod_{i<l} s_i$ .

Batch Normalisation

BatchNorm (Ioffe & Szegedy, 2015) normalises each feature map across the batch:

$\hat{x} = \frac{x - \mu_B}{\sqrt{\sigma_B^2 + \epsilon}}, \quad y = \gamma \hat{x} + \beta$

where $\mu_B, \sigma_B^2$ are computed per-channel over the batch, and $\gamma, \beta$ are learnable scale/shift parameters.

Benefits: Reduces internal covariate shift, allows higher learning rates, acts as mild regularisation.

Pooling Layers

Type	Operation	Output	Use
Max Pool	$\max$ in window	Dominant feature	Early layers
Avg Pool	mean in window	Smooth downsampling	Skip connections
Global Avg Pool	mean over entire map	Scalar per channel	Before classifier

Global Average Pooling (GAP) replaces the final flatten + fully-connected layers in modern architectures, drastically reducing parameters and enabling arbitrary input sizes.

Convolution shape and parameter calculatorpython

import torch
import torch.nn as nn

def conv_output_shape(h_in, w_in, kernel=3, stride=1, padding=0, dilation=1):
    h = (h_in + 2*padding - dilation*(kernel-1) - 1) // stride + 1
    w = (w_in + 2*padding - dilation*(kernel-1) - 1) // stride + 1
    return h, w

def count_conv_params(c_in, c_out, kernel, bias=True):
    return c_out * (c_in * kernel * kernel + int(bias))

# Example: VGG-style 3x3 conv
h, w = conv_output_shape(224, 224, kernel=3, stride=1, padding=1)
print(f"Output: {h}x{w}")   # 224x224 (same padding)

params = count_conv_params(64, 128, 3)
print(f"Params: {params:,}")  # 73,856

# Receptive field calculation
def receptive_field(layers):
    """layers: list of (kernel_size, stride) tuples"""
    rf, total_stride = 1, 1
    for k, s in layers:
        rf += (k - 1) * total_stride
        total_stride *= s
    return rf, total_stride

# AlexNet conv layers (simplified)
alex_layers = [(11,4), (5,1), (3,1), (3,1), (3,1)]
rf, stride = receptive_field(alex_layers)
print(f"AlexNet RF: {rf}, effective stride: {stride}")

ResNet: Residual Connections

The key insight of ResNet (He et al., 2016) is the residual shortcut:

$y = \mathcal{F}(x, \{W_i\}) + x$

Instead of learning the full mapping $H(x)$ , the network learns the residual $\mathcal{F}(x) = H(x) - x$ .

Why it works: When the residual mapping is zero ( $\mathcal{F}(x) = 0$ ), the layer becomes an identity — it's easy to initialise near zero. This prevents degradation and allows training extremely deep networks (50, 101, 152 layers).

BasicBlock (ResNet-18/34)

Two 3×3 convolutions with a shortcut connection:

Input → Conv3x3 → BN → ReLU → Conv3x3 → BN → (+) Input → ReLU

If the input/output dimensions differ, a 1×1 projection convolution is used in the shortcut.

Bottleneck (ResNet-50/101/152)

Three convolutions (1×1 → 3×3 → 1×1) to reduce the number of parameters:

Input → Conv1x1 → BN → ReLU → Conv3x3 → BN → ReLU → Conv1x1 → BN → (+) projected_input → ReLU

The bottleneck reduces channels before the expensive 3×3 conv and expands them after.

Architecture Evolution

Model	Year	Key Innovation	Top-1
LeNet-5	1998	First CNN, 60K params	—
AlexNet	2012	Deep + ReLU + Dropout	63.3%
VGG-16	2014	3×3 convs only	74.4%
GoogLeNet	2014	Inception module, 1×1 bottleneck	74.8%
ResNet-50	2015	Residual connections	80.9%
EfficientNet-B0	2019	Compound scaling	77.7%
ConvNeXt-T	2022	Modernised pure CNN	82.1%

Depthwise Separable Convolutions

Standard conv: $\text{Cost} = D_K^2 \cdot M \cdot N \cdot D_F^2$

Depthwise separable (depthwise + pointwise): $\text{Cost}_{DS} = D_K^2 \cdot M \cdot D_F^2 + M \cdot N \cdot D_F^2$

$\text{Reduction} = \frac{1}{N} + \frac{1}{D_K^2} \approx \frac{1}{9} \text{ for } 3\times 3 \text{ kernels}$

Used in MobileNet, Xception, and EfficientNet.

ResNet BasicBlock, Bottleneck, and SmallResNetpython

import torch
import torch.nn as nn

class BasicBlock(nn.Module):
    expansion = 1

    def __init__(self, in_ch, out_ch, stride=1):
        super().__init__()
        self.conv1 = nn.Conv2d(in_ch, out_ch, 3, stride=stride, padding=1, bias=False)
        self.bn1   = nn.BatchNorm2d(out_ch)
        self.conv2 = nn.Conv2d(out_ch, out_ch, 3, stride=1, padding=1, bias=False)
        self.bn2   = nn.BatchNorm2d(out_ch)
        self.relu  = nn.ReLU(inplace=True)

        self.shortcut = nn.Identity()
        if stride != 1 or in_ch != out_ch:
            self.shortcut = nn.Sequential(
                nn.Conv2d(in_ch, out_ch, 1, stride=stride, bias=False),
                nn.BatchNorm2d(out_ch),
            )

    def forward(self, x):
        out = self.relu(self.bn1(self.conv1(x)))
        out = self.bn2(self.conv2(out))
        return self.relu(out + self.shortcut(x))

class Bottleneck(nn.Module):
    expansion = 4

    def __init__(self, in_ch, mid_ch, stride=1):
        super().__init__()
        out_ch = mid_ch * self.expansion
        self.conv1 = nn.Conv2d(in_ch, mid_ch, 1, bias=False)
        self.bn1   = nn.BatchNorm2d(mid_ch)
        self.conv2 = nn.Conv2d(mid_ch, mid_ch, 3, stride=stride, padding=1, bias=False)
        self.bn2   = nn.BatchNorm2d(mid_ch)
        self.conv3 = nn.Conv2d(mid_ch, out_ch, 1, bias=False)
        self.bn3   = nn.BatchNorm2d(out_ch)
        self.relu  = nn.ReLU(inplace=True)
        self.shortcut = nn.Sequential(
            nn.Conv2d(in_ch, out_ch, 1, stride=stride, bias=False),
            nn.BatchNorm2d(out_ch),
        ) if (stride != 1 or in_ch != out_ch) else nn.Identity()

    def forward(self, x):
        out = self.relu(self.bn1(self.conv1(x)))
        out = self.relu(self.bn2(self.conv2(out)))
        out = self.bn3(self.conv3(out))
        return self.relu(out + self.shortcut(x))

class SmallResNet(nn.Module):
    def __init__(self, block, layers, num_classes=10):
        super().__init__()
        self.in_ch = 64
        self.conv1 = nn.Conv2d(3, 64, 3, stride=1, padding=1, bias=False)
        self.bn1   = nn.BatchNorm2d(64)
        self.relu  = nn.ReLU(inplace=True)
        self.layer1 = self._make_layer(block, 64,  layers[0], stride=1)
        self.layer2 = self._make_layer(block, 128, layers[1], stride=2)
        self.layer3 = self._make_layer(block, 256, layers[2], stride=2)
        self.avgpool = nn.AdaptiveAvgPool2d(1)
        self.fc = nn.Linear(256 * block.expansion, num_classes)
        for m in self.modules():
            if isinstance(m, nn.Conv2d):
                nn.init.kaiming_normal_(m.weight, mode='fan_out', nonlinearity='relu')

    def _make_layer(self, block, out_ch, n_blocks, stride):
        layers = [block(self.in_ch, out_ch, stride)]
        self.in_ch = out_ch * block.expansion
        for _ in range(1, n_blocks):
            layers.append(block(self.in_ch, out_ch))
        return nn.Sequential(*layers)

    def forward(self, x):
        x = self.relu(self.bn1(self.conv1(x)))
        x = self.layer1(x); x = self.layer2(x); x = self.layer3(x)
        x = self.avgpool(x).flatten(1)
        return self.fc(x)

model = SmallResNet(BasicBlock, [2, 2, 2], num_classes=10)
params = sum(p.numel() for p in model.parameters()) / 1e6
print(f"Params: {params:.2f}M")  # ~0.96M
print(model(torch.randn(2, 3, 32, 32)).shape)  # (2, 10)

Knowledge check

A Conv2d layer with in_channels=64, out_channels=128, kernel_size=3, padding=1, stride=1 processes a 56×56 feature map. What is the output shape?

Knowledge check

What problem does the residual (skip) connection in ResNet solve?

Knowledge check

What is the parameter reduction factor of depthwise separable convolution vs standard convolution for a 3×3 kernel?

← Computer Vision