Deep Neural Networks

From Perceptron to Deep Network

A perceptron computes a weighted sum of inputs and applies a threshold: $y = \begin{cases} 1 & \text{if } \mathbf{w}^T \mathbf{x} + b \geq 0 \\ 0 & \text{otherwise} \end{cases}$

Stack multiple layers of neurons → Multilayer Perceptron (MLP):

Input Layer → Hidden Layer 1 → Hidden Layer 2 → ... → Output Layer
  (features)     (abstractions)   (higher abstractions)   (predictions)

Each layer transforms its input: $\mathbf{h}^{(l)} = g\left(W^{(l)} \mathbf{h}^{(l-1)} + \mathbf{b}^{(l)}\right)$

where $g$ is a non-linear activation function.

Universal Approximation Theorem

An MLP with a single hidden layer of sufficient width can approximate any continuous function to arbitrary precision. Depth (many layers) is more parameter-efficient than width alone.

Activation Functions

Without non-linearity, a deep network collapses to a single linear transformation. Activation functions introduce non-linearity:

Function	Formula	Range	Use Case
Sigmoid	$\frac{1}{1+e^{-x}}$	(0, 1)	Binary output layer
Tanh	$\frac{e^x - e^{-x}}{e^x + e^{-x}}$	(-1, 1)	Hidden layers (old)
ReLU	$\max(0, x)$	$[0, \infty)$	Default hidden layer choice
Leaky ReLU	$\max(\alpha x, x)$	$(-\infty, \infty)$	Avoids dying ReLU
GELU	$x \cdot \Phi(x)$	$(-\infty, \infty)$	Transformers, BERT, GPT
Softmax	$\frac{e^{x_i}}{\sum_j e^{x_j}}$	(0, 1), sum=1	Multi-class output layer

Dying ReLU problem: if a neuron's input is always negative, its gradient is always 0 — it never learns. Leaky ReLU, ELU, and GELU avoid this.

Backpropagation

Backpropagation efficiently computes gradients of the loss with respect to every parameter using the chain rule.

Forward Pass

Compute predictions layer by layer, storing intermediate activations.

Backward Pass

Starting from the loss, propagate gradients backward:

$\frac{\partial \mathcal{L}}{\partial W^{(l)}} = \frac{\partial \mathcal{L}}{\partial \mathbf{h}^{(l)}} \cdot \frac{\partial \mathbf{h}^{(l)}}{\partial W^{(l)}}$

Vanishing Gradient Problem

In deep networks, gradients shrink exponentially as they propagate backward through sigmoid/tanh layers: $\frac{\partial \sigma}{\partial x}\bigg|_{x=\pm 5} \approx 0$

Solutions:

• Use ReLU / GELU activations
• Batch Normalization
• Residual connections (ResNet)
• Careful weight initialization (Xavier, He)

Regularization Techniques

Dropout

During training, randomly set a fraction $p$ of neurons to zero at each forward pass. Forces the network to learn redundant representations.

• Train time: each neuron active with probability $1-p$; multiply activations by $\frac{1}{1-p}$ at test time (inverted dropout)
• Typical $p$: 0.2–0.5 for hidden layers; lower for convolutional layers

Batch Normalization

Normalize the inputs to each layer across the mini-batch: $\hat{x}_i = \frac{x_i - \mu_B}{\sqrt{\sigma_B^2 + \epsilon}}; \quad y_i = \gamma \hat{x}_i + \beta$

Benefits:

• Reduces internal covariate shift → faster, more stable training
• Acts as regularizer → can reduce or eliminate dropout
• Allows higher learning rates
• Almost always used in modern architectures

Weight Initialization

• Xavier / Glorot: $W \sim \mathcal{N}(0, \sqrt{2/(n_{in}+n_{out})})$ — good for sigmoid/tanh
• He (Kaiming): $W \sim \mathcal{N}(0, \sqrt{2/n_{in}})$ — good for ReLU

import torch
import torch.nn as nn
import torch.optim as optim
from torch.utils.data import TensorDataset, DataLoader
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
import numpy as np

# Data preparation
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
scaler = StandardScaler()
X_train = scaler.fit_transform(X_train)
X_test = scaler.transform(X_test)

# Convert to PyTorch tensors
X_tr = torch.FloatTensor(X_train)
y_tr = torch.FloatTensor(y_train)
X_te = torch.FloatTensor(X_test)
y_te = torch.FloatTensor(y_test)

train_loader = DataLoader(TensorDataset(X_tr, y_tr), batch_size=32, shuffle=True)

# Define MLP with BatchNorm + Dropout
class MLP(nn.Module):
    def __init__(self, input_dim, hidden_dims, dropout=0.3):
        super().__init__()
        layers = []
        prev_dim = input_dim
        for dim in hidden_dims:
            layers += [
                nn.Linear(prev_dim, dim),
                nn.BatchNorm1d(dim),
                nn.ReLU(),
                nn.Dropout(dropout),
            ]
            prev_dim = dim
        layers.append(nn.Linear(prev_dim, 1))
        self.net = nn.Sequential(*layers)

    def forward(self, x):
        return self.net(x).squeeze(1)

model = MLP(input_dim=30, hidden_dims=[128, 64, 32], dropout=0.3)
criterion = nn.BCEWithLogitsLoss()
optimizer = optim.Adam(model.parameters(), lr=1e-3, weight_decay=1e-4)
scheduler = optim.lr_scheduler.StepLR(optimizer, step_size=20, gamma=0.5)

# Training loop
for epoch in range(60):
    model.train()
    for X_batch, y_batch in train_loader:
        optimizer.zero_grad()
        loss = criterion(model(X_batch), y_batch)
        loss.backward()
        optimizer.step()
    scheduler.step()

# Evaluation
model.eval()
with torch.no_grad():
    logits = model(X_te)
    preds = (torch.sigmoid(logits) > 0.5).float()
    acc = (preds == y_te).float().mean()
    print(f"Test Accuracy: {acc:.4f}")

Summary

• MLPs stack linear transformations with non-linear activations to approximate complex functions
• ReLU / GELU solve the vanishing gradient problem of sigmoid/tanh
• Backpropagation efficiently computes gradients via the chain rule
• Dropout prevents co-adaptation of neurons; Batch Normalization stabilizes training
• He initialization + Adam optimizer + learning rate scheduling is the reliable training recipe
• Residual connections (skip connections) enable training very deep networks (100+ layers)

Next: Convolutional Neural Networks (CNNs) — specialized architectures for image and spatial data.