Diffusion Models Deep Dive

The Core Idea: Destruction and Reconstruction

Diffusion models learn to generate data by learning to reverse a noise-adding process.

Forward process: gradually corrupt a real image $x_0$ into pure Gaussian noise over $T$ steps.

Reverse process: learn a neural network that denoises step-by-step, reconstructing a clean image from noise.

At inference time, sample pure noise and apply the learned reverse process → new image.

Why Diffusion Beat GANs

Property	GANs	Diffusion
Training stability	Prone to mode collapse	Stable (regression objective)
Sample diversity	Often limited	Excellent
Sample quality	Sharp but sometimes artifacts	State of the art
Controllability	Limited	CFG, ControlNet, IP-Adapter
Likelihood	Not directly available	Tractable (ELBO)

Diffusion models (DALL-E 3, Stable Diffusion, Imagen, Midjourney v6) now dominate image generation.

Forward Process (Noise Schedule)

Define a variance schedule $\beta_1 < \beta_2 < \ldots < \beta_T$ (e.g., linearly from $10^{-4}$ to $0.02$).

The forward process adds noise step by step:

$q(x_t \mid x_{t-1}) = \mathcal{N}(x_t;\ \sqrt{1 - \beta_t}\, x_{t-1},\ \beta_t I)$

Key shortcut: sample $x_t$ at any step $t$ directly from $x_0$ in closed form:

$q(x_t \mid x_0) = \mathcal{N}(x_t;\ \sqrt{\bar{\alpha}_t}\, x_0,\ (1 - \bar{\alpha}_t) I)$

$x_t = \sqrt{\bar{\alpha}_t}\, x_0 + \sqrt{1 - \bar{\alpha}_t}\, \varepsilon, \quad \varepsilon \sim \mathcal{N}(0, I)$

where $\alpha_t = 1 - \beta_t$ and $\bar{\alpha}_t = \prod_{s=1}^t \alpha_s$.

As $t \to T$: $\bar{\alpha}_T \approx 0$ → $x_T \approx \mathcal{N}(0, I)$ (pure noise).

Noise Schedules

Schedule	Formula	Used In
Linear	$\beta_t =$ linear	DDPM original
Cosine	$\bar{\alpha}_t = \cos^2\!\left(\frac{t/T + s}{1+s} \cdot \frac{\pi}{2}\right)$	Improved DDPM — avoids sudden noise at end
Sigmoid / EDM	Continuous formulations	SDXL, EDM (Karras et al.)

Reverse Process and Training Objective

The reverse process is modeled as:

$p_\theta(x_{t-1} \mid x_t) = \mathcal{N}(x_{t-1};\ \mu_\theta(x_t, t),\ \Sigma_\theta(x_t, t))$

A U-Net (or Diffusion Transformer) $\varepsilon_\theta(x_t, t)$ is trained to predict the noise $\varepsilon$ added at step $t$.

ELBO → Simplified Loss

The full ELBO objective simplifies to:

$\mathcal{L}_{\text{simple}} = \mathbb{E}_{t, x_0, \varepsilon}\left[\|\varepsilon - \varepsilon_\theta(\sqrt{\bar{\alpha}_t}\, x_0 + \sqrt{1 - \bar{\alpha}_t}\, \varepsilon,\ t)\|^2\right]$

In plain English: pick a random timestep $t$, add that much noise to the image, ask the network to predict the noise, minimize prediction error.

This is just denoising score matching — the network learns the gradient of the data log-density.

Inference (Ancestral Sampling)

x_T ~ N(0, I)
for t = T, T-1, ..., 1:
    z ~ N(0, I)  if t > 1 else z = 0
    predicted_noise = ε_θ(x_t, t)
    x_{t-1} = (1/√α_t) * (x_t - β_t/√(1-ᾱ_t) * predicted_noise) + √β_t * z
return x_0

Problem: requires $T = 1000$ network forward passes — very slow.

DDIM — Denoising Diffusion Implicit Models

Song et al. (2021) reinterpret the forward process as non-Markovian, allowing much larger step sizes.

DDIM update rule (deterministic, $\eta = 0$):

$x_{t-1} = \sqrt{\bar{\alpha}_{t-1}}\underbrace{\left(\frac{x_t - \sqrt{1-\bar{\alpha}_t}\,\varepsilon_\theta(x_t,t)}{\sqrt{\bar{\alpha}_t}}\right)}_{\text{predicted }x_0} + \underbrace{\sqrt{1-\bar{\alpha}_{t-1}}\,\varepsilon_\theta(x_t,t)}_{\text{direction to }x_t}$

Why DDIM Matters

• 10–50× speedup: generate in 20–50 steps instead of 1000
• Deterministic: same noise → same image (reproducible)
• Interpolation in noise space: latent space traversal is meaningful
• Used by all modern samplers: PLMS, DPM-Solver, DPM-Solver++, UniPC

Modern Sampler Comparison

Sampler	Steps Needed	Quality	Speed
DDPM	1000	Good	Very slow
DDIM	50	Good	Fast
DPM-Solver++	20	Excellent	Very fast
LCM (Latent Consistency)	4–8	Good	Extremely fast

Classifier-Free Guidance (CFG)

Ho & Salimans (2022) — the technique behind prompt adherence in Stable Diffusion, DALL-E 3, and Imagen.

The Idea

Train a single conditional model $\varepsilon_\theta(x_t, t, c)$ that also handles the unconditional case by randomly dropping the condition $c$ during training (replaced with a null token $\emptyset$).

At inference, interpolate between conditional and unconditional predictions:

$\tilde{\varepsilon}_\theta(x_t, t, c) = \varepsilon_\theta(x_t, t, \emptyset) + w \cdot \underbrace{(\varepsilon_\theta(x_t, t, c) - \varepsilon_\theta(x_t, t, \emptyset))}_{\text{guidance direction}}$

• $w$ = guidance scale (CFG scale) — typically 7–12 for text-to-image
• $w = 0$: unconditional (diverse but may ignore prompt)
• $w > 1$: trades diversity for prompt adherence
• $w$ too high: oversaturation, artifacts

What $c$ Can Be

• Text embeddings (CLIP, T5, BERT)
• Class labels
• Image embeddings (IP-Adapter)
• Structural conditions (depth, pose, edge via ControlNet)

import torch
import torch.nn as nn
import torch.nn.functional as F
import numpy as np

# ---- Noise schedule ----
def cosine_beta_schedule(T, s=0.008):
    steps = torch.arange(T + 1, dtype=torch.float64)
    alphas_cumprod = torch.cos(((steps / T) + s) / (1 + s) * np.pi / 2) ** 2
    alphas_cumprod = alphas_cumprod / alphas_cumprod[0]
    betas = 1 - (alphas_cumprod[1:] / alphas_cumprod[:-1])
    return torch.clamp(betas, 0, 0.999).float()

T = 1000
betas = cosine_beta_schedule(T)
alphas = 1 - betas
alphas_cumprod = torch.cumprod(alphas, dim=0)       # ᾱ_t
sqrt_alphas_cumprod = alphas_cumprod.sqrt()
sqrt_one_minus_alphas_cumprod = (1 - alphas_cumprod).sqrt()

# ---- Forward diffusion: sample x_t given x_0 ----
def q_sample(x0, t, noise=None):
    if noise is None:
        noise = torch.randn_like(x0)
    sa = sqrt_alphas_cumprod[t].view(-1, 1, 1, 1)
    sm = sqrt_one_minus_alphas_cumprod[t].view(-1, 1, 1, 1)
    return sa * x0 + sm * noise, noise

# ---- Tiny U-Net (stand-in for real U-Net) ----
class TinyDenoiser(nn.Module):
    def __init__(self, dim=32):
        super().__init__()
        self.time_emb = nn.Embedding(T, dim)
        self.net = nn.Sequential(
            nn.Conv2d(1 + dim, 64, 3, padding=1), nn.SiLU(),
            nn.Conv2d(64, 64, 3, padding=1),       nn.SiLU(),
            nn.Conv2d(64, 1, 1),
        )
    def forward(self, x, t):
        t_emb = self.time_emb(t)[:, :, None, None].expand(-1, -1, x.shape[2], x.shape[3])
        return self.net(torch.cat([x, t_emb], dim=1))

# ---- Training step ----
def train_step(model, optimizer, x0):
    B = x0.shape[0]
    t = torch.randint(0, T, (B,))
    xt, noise = q_sample(x0, t)
    pred_noise = model(xt, t)
    loss = F.mse_loss(pred_noise, noise)
    optimizer.zero_grad(); loss.backward(); optimizer.step()
    return loss.item()

# ---- DDIM Sampling (50 steps) ----
@torch.no_grad()
def ddim_sample(model, shape, steps=50, eta=0.0):
    device = next(model.parameters()).device
    # Select subset of timesteps
    step_seq = torch.linspace(0, T - 1, steps).long()
    step_seq_prev = torch.cat([torch.tensor([-1]), step_seq[:-1]])

    x = torch.randn(shape, device=device)
    for t, t_prev in zip(reversed(step_seq), reversed(step_seq_prev)):
        t_batch = torch.full((shape[0],), t, device=device, dtype=torch.long)
        pred_noise = model(x, t_batch)

        ac = alphas_cumprod[t]
        ac_prev = alphas_cumprod[t_prev] if t_prev >= 0 else torch.tensor(1.0)

        x0_pred = (x - (1 - ac).sqrt() * pred_noise) / ac.sqrt()
        x0_pred = x0_pred.clamp(-1, 1)

        sigma = eta * ((1 - ac_prev) / (1 - ac) * (1 - ac / ac_prev)).sqrt()
        direction = (1 - ac_prev - sigma**2).clamp(0).sqrt() * pred_noise
        noise = torch.randn_like(x) if eta > 0 else 0
        x = ac_prev.sqrt() * x0_pred + direction + sigma * noise

    return x

print("DDPM training loop and DDIM sampler ready.")
print(f"Forward: T={T} steps | DDIM inference: 50 steps ({T//50}x speedup)")

LoRA for Diffusion Models

Low-Rank Adaptation applied to diffusion models (U-Net attention layers) enables:

• Style LoRAs: teach the model a new art style from ~20 images
• Subject LoRAs (DreamBooth + LoRA): teach the model a specific person or object
• Concept LoRAs: add characters, products, or visual concepts

How It Works

Freeze the base U-Net weights $W_0$. Add a low-rank update:

$W = W_0 + \Delta W = W_0 + B \cdot A, \quad B \in \mathbb{R}^{d \times r},\ A \in \mathbb{R}^{r \times k},\ r \ll \min(d,k)$

Train only $A$ and $B$ on the target images using the standard DDPM loss.

Method	Parameters Trained	Images Needed	Use Case
Full fine-tune	All U-Net weights	100s–1000s	Large style shifts
DreamBooth	All U-Net + text encoder	5–30	Specific subject
LoRA	Rank-4 to rank-64 deltas	10–100	Style, subject, concept
Textual Inversion	New text embedding only	3–10	Simple concept binding

Latent Diffusion (Stable Diffusion)

SD runs diffusion in latent space (not pixel space):

Image → VAE Encoder → Latent z (64×64) → Diffusion → Latent → VAE Decoder → Image

• 4× compression per dimension → 16× fewer pixels to denoise → faster and cheaper
• SDXL uses two CLIP text encoders and a larger U-Net (2.6B params)
• SD3 / FLUX use Diffusion Transformers (DiT) instead of U-Net

Summary

• DDPM: learns to reverse Gaussian noise addition; simplified loss = predict the noise at each step
• Key insight: $x_t$ can be sampled from $x_0$ in closed form using $\bar{\alpha}_t$ — enables efficient training
• DDIM: non-Markovian reformulation enabling 20–50 step inference (vs 1000 for DDPM)
• CFG: train conditional + unconditional jointly, extrapolate at inference to control prompt adherence ($w$ scale)
• Latent Diffusion (SD): run diffusion in compressed VAE latent space for speed; SDXL and FLUX extend this
• LoRA / DreamBooth: fine-tune diffusion models for custom styles, subjects, or concepts in hours on a single GPU

Next: Multimodal AI — DALL-E 3, Gemini, GPT-4V, and image-text retrieval at scale.