Diffusion Models — AI Tutorials — Subhendu Datta Bhowmik

The Diffusion Framework

DDPM (Ho et al., 2020) defines two Markov chains:

Forward Process (adding noise)

Starting from real data $x_0$ , progressively add Gaussian noise over $T$ steps:

$q(x_t | x_{t-1}) = \mathcal{N}(x_t; \sqrt{1 - \beta_t} x_{t-1}, \beta_t I)$

Using $\bar{\alpha}_t = \prod_{s=1}^t (1-\beta_s)$ , we can directly sample $x_t$ from $x_0$ :

$x_t = \sqrt{\bar{\alpha}_t} x_0 + \sqrt{1 - \bar{\alpha}_t} \epsilon, \quad \epsilon \sim \mathcal{N}(0, I)$

Training Objective

Train a noise predictor $\epsilon_\theta$ :

$\mathcal{L} = \mathbb{E}_{t, x_0, \epsilon}\left[\|\epsilon - \epsilon_\theta(\sqrt{\bar{\alpha}_t} x_0 + \sqrt{1 - \bar{\alpha}_t} \epsilon,\; t)\|^2\right]$

DDIM: Fast Sampling

DDIM (Song et al., 2020) reformulates the reverse process as non-Markovian, enabling deterministic sampling (η=0) with only 10–50 steps instead of 1000, using the same trained model.

$x_{t-1} = \sqrt{\bar{\alpha}_{t-1}} \hat{x}_0 + \sqrt{1-\bar{\alpha}_{t-1}} \cdot \epsilon_\theta(x_t, t)$

Stable Diffusion: Latent Diffusion

Running diffusion in pixel space is expensive. Stable Diffusion (Rombach et al., 2022) operates in a compressed latent space:

VAE encoder compresses $512\times512\times3 \to 64\times64\times4$ (8× reduction)
Diffusion operates on latents (64× fewer pixels)
Text conditioning via CLIP text encoder cross-attention in the U-Net
VAE decoder reconstructs the image

Classifier-Free Guidance (CFG)

During training, randomly drop conditioning with probability ~10%. At inference, blend conditional and unconditional score estimates:

$\tilde{\epsilon}_\theta(x_t, c) = \epsilon_\theta(x_t, \emptyset) + w \cdot (\epsilon_\theta(x_t, c) - \epsilon_\theta(x_t, \emptyset))$

Higher guidance scale $w$ → stronger prompt adherence (typical: $w \in [7, 15]$ ).

DDPM noise schedule and forward processpython

import torch
import torch.nn.functional as F

def cosine_beta_schedule(timesteps: int, s=0.008):
    steps = torch.arange(timesteps + 1, dtype=torch.float64)
    alphas_cumprod = torch.cos(((steps / timesteps) + s) / (1 + s) * torch.pi / 2) ** 2
    alphas_cumprod /= alphas_cumprod[0]
    betas = 1 - (alphas_cumprod[1:] / alphas_cumprod[:-1])
    return betas.clamp(0, 0.999).float()

class DiffusionSchedule:
    def __init__(self, timesteps=1000):
        self.T = timesteps
        betas = cosine_beta_schedule(timesteps)
        alphas = 1.0 - betas
        alphas_cumprod = alphas.cumprod(dim=0)
        self.sqrt_ac   = alphas_cumprod.sqrt()
        self.sqrt_1mac = (1 - alphas_cumprod).sqrt()
        self.betas = betas
        self.sqrt_recip_alphas = (1.0 / alphas).sqrt()
        self.post_var = betas * (1 - alphas_cumprod.roll(1)) / (1 - alphas_cumprod)

    def q_sample(self, x0, t, noise=None):
        """Sample x_t given x_0."""
        if noise is None: noise = torch.randn_like(x0)
        sa  = self.sqrt_ac[t].view(-1, 1, 1, 1).to(x0.device)
        s1a = self.sqrt_1mac[t].view(-1, 1, 1, 1).to(x0.device)
        return sa * x0 + s1a * noise, noise

def train_ddpm(model, dataloader, schedule, epochs=100, lr=1e-4, device='cuda'):
    model = model.to(device)
    optimizer = torch.optim.AdamW(model.parameters(), lr=lr)
    for epoch in range(epochs):
        for x0, _ in dataloader:
            x0 = x0.to(device)
            t = torch.randint(0, schedule.T, (x0.size(0),), device=device)
            xt, noise = schedule.q_sample(x0, t)
            loss = F.mse_loss(model(xt, t), noise)
            optimizer.zero_grad(); loss.backward()
            torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
            optimizer.step()
        print(f"Epoch {epoch+1}  loss={loss.item():.4f}")

@torch.no_grad()
def ddim_sample(model, schedule, img_shape, ddim_steps=50, eta=0.0, device='cuda'):
    """DDIM sampling — much faster than DDPM."""
    model.eval()
    ac = schedule.sqrt_ac.to(device) ** 2   # alphas_cumprod
    step_size = schedule.T // ddim_steps
    timesteps = list(reversed(range(0, schedule.T, step_size)))
    x = torch.randn(img_shape, device=device)

    for i, t_val in enumerate(timesteps):
        t = torch.full((img_shape[0],), t_val, device=device, dtype=torch.long)
        pred_noise = model(x, t)
        at = ac[t_val]
        x0_pred = (x - (1-at).sqrt() * pred_noise) / at.sqrt()
        x0_pred = x0_pred.clamp(-1, 1)
        if i < len(timesteps) - 1:
            at_prev = ac[timesteps[i+1]]
            sigma = eta * ((1-at_prev)/(1-at) * (1-at/at_prev)).sqrt()
            x = at_prev.sqrt()*x0_pred + (1-at_prev-sigma**2).sqrt()*pred_noise
            if eta > 0: x = x + sigma * torch.randn_like(x)
        else:
            x = x0_pred
    return x.clamp(-1, 1)

Stable Diffusion with HuggingFace diffuserspython

# pip install diffusers transformers accelerate
from diffusers import StableDiffusionPipeline, DPMSolverMultistepScheduler
import torch

# Text-to-image
pipe = StableDiffusionPipeline.from_pretrained(
    "runwayml/stable-diffusion-v1-5", torch_dtype=torch.float16,
)
pipe.scheduler = DPMSolverMultistepScheduler.from_config(pipe.scheduler.config)
pipe = pipe.to("cuda")

image = pipe(
    prompt="a photorealistic tabby cat on a wooden desk, soft morning light",
    negative_prompt="blurry, low quality, cartoon",
    num_inference_steps=25,
    guidance_scale=7.5,
).images[0]
image.save("cat.png")

# Image-to-image
from diffusers import StableDiffusionImg2ImgPipeline
from PIL import Image

img2img = StableDiffusionImg2ImgPipeline.from_pretrained(
    "runwayml/stable-diffusion-v1-5", torch_dtype=torch.float16
).to("cuda")

init_image = Image.open("sketch.png").convert("RGB").resize((512, 512))
result = img2img(
    prompt="a detailed oil painting of a medieval castle",
    image=init_image,
    strength=0.75,       # 0=no change, 1=full generation
    guidance_scale=7.5,
).images[0]

# Fine-tuning with LoRA (DreamBooth):
# python train_dreambooth_lora.py \
#   --pretrained_model_name_or_path="runwayml/stable-diffusion-v1-5" \
#   --instance_data_dir="./my_images" \
#   --instance_prompt="a photo of sks cat" \
#   --rank=4 --num_train_epochs=100

Knowledge check

In DDPM, what does the model ε_θ(x_t, t) predict?

Knowledge check

What is the key advantage of DDIM over DDPM?

Knowledge check

In classifier-free guidance (CFG), a higher guidance scale w means:

← Computer Vision