Different Generative Models

What Is a Generative Model?

A generative model learns the underlying distribution of training data so it can produce new samples that look like they came from that distribution. In contrast, a discriminative model only learns to distinguish between classes.

Generative AI has exploded because of four key breakthroughs, each representing a distinct model family:

Model Family	Core Idea	Famous Examples
GANs	Two networks compete	StyleGAN, BigGAN
VAEs	Encode to latent space, decode	DALL-E (v1), VQ-VAE
Diffusion Models	Learn to reverse noise	Stable Diffusion, DALL-E 3, Midjourney
Autoregressive	Predict the next token	GPT-4, Claude, Gemini, LLaMA

Each family has different strengths, failure modes, and computational profiles.

Generative Adversarial Networks (GANs)

Introduced by Ian Goodfellow in 2014, GANs use a minimax game between two networks:

• Generator (G): Takes random noise z and produces fake samples G(z)
• Discriminator (D): Tries to distinguish real samples from G(z)

The generator wins when the discriminator can't tell its outputs from real data.

GAN Training Objective

$\min_G \max_D \mathbb{E}_{x \sim p_{data}}[\log D(x)] + \mathbb{E}_{z \sim p_z}[\log(1 - D(G(z)))]$

Strengths

• Extremely sharp, high-fidelity image generation
• Fast inference (single forward pass through G)
• Great for specific domains: faces (StyleGAN), super-resolution

Weaknesses

• Mode collapse: the generator produces limited variety
• Training instability: requires careful hyperparameter tuning
• No likelihood estimation: can't measure how probable a sample is

import torch
import torch.nn as nn

# Generator: noise → fake image
class Generator(nn.Module):
    def __init__(self, latent_dim=100, img_dim=784):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(latent_dim, 256), nn.LeakyReLU(0.2),
            nn.Linear(256, 512),       nn.LeakyReLU(0.2),
            nn.Linear(512, img_dim),   nn.Tanh(),
        )
    def forward(self, z):
        return self.net(z)

# Discriminator: image → real/fake probability
class Discriminator(nn.Module):
    def __init__(self, img_dim=784):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(img_dim, 512), nn.LeakyReLU(0.2),
            nn.Linear(512, 256),     nn.LeakyReLU(0.2),
            nn.Linear(256, 1),       nn.Sigmoid(),
        )
    def forward(self, x):
        return self.net(x)

G, D = Generator(), Discriminator()
opt_G = torch.optim.Adam(G.parameters(), lr=2e-4)
opt_D = torch.optim.Adam(D.parameters(), lr=2e-4)
criterion = nn.BCELoss()

def train_step(real_imgs):
    batch = real_imgs.size(0)
    real_labels = torch.ones(batch, 1)
    fake_labels = torch.zeros(batch, 1)

    # Train Discriminator
    z = torch.randn(batch, 100)
    fake_imgs = G(z).detach()
    loss_D = criterion(D(real_imgs), real_labels) + criterion(D(fake_imgs), fake_labels)
    opt_D.zero_grad(); loss_D.backward(); opt_D.step()

    # Train Generator
    z = torch.randn(batch, 100)
    loss_G = criterion(D(G(z)), real_labels)  # fool the discriminator
    opt_G.zero_grad(); loss_G.backward(); opt_G.step()

    return loss_D.item(), loss_G.item()

Variational Autoencoders (VAEs)

VAEs (Kingma & Welling, 2013) take a probabilistic approach. They learn a latent space where similar data points cluster together, enabling smooth interpolation and controlled generation.

Architecture

1. Encoder maps input X → mean μ and variance σ² of a latent Gaussian
2. Reparameterization trick: z = μ + σ · ε (where ε ~ N(0,1)) makes it differentiable
3. Decoder maps z back to the data space

ELBO Loss

$\mathcal{L} = \mathbb{E}[\log p(x|z)] - \text{KL}(q(z|x) \| p(z))$

• Reconstruction term: decoded output should match input
• KL term: latent distribution should stay close to N(0,1)

Strengths

• Smooth, structured latent space (great for interpolation)
• Principled probabilistic framework
• Good for anomaly detection

Weaknesses

• Outputs are often blurry compared to GANs or diffusion models
• Posterior collapse can occur with powerful decoders

Diffusion Models

Diffusion models (Ho et al., 2020) are now the dominant approach for image generation. They work by:

1. Forward process: Gradually add Gaussian noise to data over T steps until it becomes pure noise
2. Reverse process: Train a neural network to predict and remove the noise at each step

Why Diffusion Won

Property	GAN	VAE	Diffusion
Sample quality	Excellent	Good	State-of-the-art
Diversity	Poor (mode collapse)	Good	Excellent
Training stability	Hard	Moderate	Stable
Inference speed	Fast	Fast	Slow (many steps)
Controllability	Limited	Moderate	Excellent (classifier-free guidance)

Classifier-Free Guidance (CFG)

The key innovation that made text-to-image work: train with and without the condition (text prompt), then at inference mix conditional and unconditional predictions with a guidance scale. Higher guidance = more prompt adherence, less diversity.

Stable Diffusion, DALL-E 3, and Midjourney all use diffusion models with latent compression (operating in a compressed latent space, not pixel space, for efficiency).

Autoregressive Models

Autoregressive models — the family powering GPT, Claude, Gemini, and LLaMA — generate sequences by predicting the next token given all previous tokens:

$P(x_1, x_2, ..., x_n) = \prod_{i=1}^{n} P(x_i | x_1, ..., x_{i-1})$

For text, tokens are sub-word pieces (typically 3–4 characters each). The model:

1. Embeds each token into a high-dimensional vector
2. Passes through transformer layers (self-attention + FFN)
3. Projects back to vocabulary size and samples the next token

Key Properties

• No mode collapse: can produce unlimited diversity
• Exact likelihood: log-likelihood is tractable
• Autoregressive slowness: must generate token-by-token
• Context window: limited by attention complexity (though modern models handle 128K–1M tokens)

Sampling Strategies

• Greedy: always pick the highest probability token (deterministic, often repetitive)
• Temperature: divide logits by T before softmax (T<1 = sharper, T>1 = more random)
• Top-p (nucleus): sample from the smallest set of tokens covering p% of probability mass
• Top-k: sample from the top-k tokens only

import anthropic

client = anthropic.Anthropic()

# Standard generation (autoregressive, token by token)
response = client.messages.create(
    model="claude-sonnet-4-6",
    max_tokens=256,
    messages=[{"role": "user", "content": "Write a haiku about neural networks"}],
)
print(response.content[0].text)

# Streaming: see tokens as they are generated
with client.messages.stream(
    model="claude-sonnet-4-6",
    max_tokens=256,
    messages=[{"role": "user", "content": "Explain diffusion models in one paragraph"}],
) as stream:
    for text in stream.text_stream:
        print(text, end="", flush=True)
print()

# Temperature control (via API — Claude uses its own calibration)
# Most APIs expose temperature as a top-level parameter:
# temperature=0.0  → deterministic, most likely token at each step
# temperature=1.0  → default sampling
# temperature=1.5  → more creative/random

Summary

• GANs: adversarial training, fast inference, sharp outputs, but unstable and prone to mode collapse
• VAEs: principled latent space, smooth interpolation, but blurry outputs
• Diffusion models: state-of-the-art image quality with excellent diversity, but slow inference
• Autoregressive models: the backbone of LLMs, exact likelihoods, sequential generation

In the next chapter, we'll go deep on the Transformer architecture — the engine that powers modern autoregressive models and has also revolutionized diffusion model conditioning.