Multimodal AI

The Multimodal Revolution

For most of ML history, models were unimodal — they processed either images or text, never both jointly. The multimodal era changed this with two key advances:

1. Contrastive pretraining (CLIP, 2021): align image and text in a shared embedding space
2. Vision-Language Models (GPT-4V, Gemini, LLaVA, 2023): connect vision encoders to powerful LLMs

Today, frontier models (GPT-4o, Gemini 1.5, Claude 3.x) process images, audio, video, and text natively in a single model.

Why Multimodal Matters

Use Case	Example
Visual QA	"What is wrong with this X-ray?"
Document understanding	Extract tables from PDF screenshots
Image-text retrieval	Find all images matching a text query
Text-to-image generation	DALL-E 3, Midjourney, Stable Diffusion
Chart / diagram analysis	"Summarize this bar chart"
Code from screenshots	"Convert this wireframe to React"

CLIP: Contrastive Image-Text Pretraining

Architecture

• Image encoder: ViT (Vision Transformer) or ResNet — produces image embedding $\mathbf{v}_i$
• Text encoder: Transformer — produces text embedding $\mathbf{u}_i$
• Both projected to a shared $d$-dimensional space and L2-normalized

Training Objective (InfoNCE / Contrastive Loss)

Given a batch of $N$ (image, text) pairs:

$\mathcal{L} = -\frac{1}{2N}\sum_{i=1}^N \left[\log \frac{\exp(\mathbf{v}_i \cdot \mathbf{u}_i / \tau)}{\sum_j \exp(\mathbf{v}_i \cdot \mathbf{u}_j / \tau)} + \log \frac{\exp(\mathbf{u}_i \cdot \mathbf{v}_i / \tau)}{\sum_j \exp(\mathbf{u}_i \cdot \mathbf{v}_j / \tau)}\right]$

where $\tau$ is a learnable temperature. Pushes matching pairs to cosine similarity 1 and non-matching pairs to 0.

What CLIP Enables

Application	How
Zero-shot classification	Embed class names as text; classify by nearest text embedding
Image search	Embed query text; retrieve nearest image embeddings
Text-to-image guidance	CLIP score used in DALL-E, Imagen, Stable Diffusion
Image captioning	Use image embedding as prefix to a language model
Cross-modal retrieval	Find images given text or text given images

CLIP Limitations

• Struggles with fine-grained counting ("3 red balls, 2 blue ones")
• Weak on spatial relationships ("object A is to the left of B")
• Biases from web-scale data
• Does not generate — only embeds

from PIL import Image
import torch
import open_clip

# ---- Load CLIP model ----
model, _, preprocess = open_clip.create_model_and_transforms(
    'ViT-B-32', pretrained='openai'
)
tokenizer = open_clip.get_tokenizer('ViT-B-32')
model.eval()

# ---- Zero-shot image classification ----
# Load a sample image (replace with any image path)
image = preprocess(Image.new('RGB', (224, 224), color='blue')).unsqueeze(0)

class_names = ['a cat', 'a dog', 'a blue rectangle', 'a red car', 'a forest']
text_tokens = tokenizer(class_names)

with torch.no_grad():
    image_features = model.encode_image(image)        # (1, 512)
    text_features  = model.encode_text(text_tokens)   # (5, 512)

    # Normalize
    image_features = image_features / image_features.norm(dim=-1, keepdim=True)
    text_features  = text_features  / text_features.norm(dim=-1, keepdim=True)

    # Cosine similarities
    logits = (image_features @ text_features.T).squeeze(0)  # (5,)
    probs  = logits.softmax(dim=-1)

print("Zero-shot classification:")
for name, p in zip(class_names, probs):
    print(f"  {name:<25} {p.item():.3f}")

# ---- Image-text retrieval ----
# Simulate a database of image embeddings
import torch
torch.manual_seed(0)
N_images = 1000
db_image_embeddings = torch.randn(N_images, 512)
db_image_embeddings = db_image_embeddings / db_image_embeddings.norm(dim=-1, keepdim=True)

query = "a sunset over the ocean"
query_tokens = tokenizer([query])
with torch.no_grad():
    query_emb = model.encode_text(query_tokens)
    query_emb = query_emb / query_emb.norm(dim=-1, keepdim=True)

scores = (query_emb @ db_image_embeddings.T).squeeze(0)  # (N_images,)
top5_indices = scores.topk(5).indices.tolist()
top5_scores  = scores.topk(5).values.tolist()

print(f"
Top-5 images for query: '{query}'")
for rank, (idx, score) in enumerate(zip(top5_indices, top5_scores), 1):
    print(f"  Rank {rank}: image_id={idx:4d}, similarity={score:.4f}")

# ---- Compute CLIP score for a text-image pair ----
clip_score = (image_features @ query_emb.T).item()
print(f"
CLIP score (image vs query): {clip_score:.4f}")

Vision-Language Models: Connecting Vision to LLMs

Modern VLMs connect a pretrained vision encoder to a powerful LLM via a projection layer.

General Architecture

Image → Vision Encoder (ViT/CLIP) → Projection Layer → Visual Tokens
                                                              ↓
Text tokens ──────────────────────────────────────────→ LLM Decoder → Response

Key Models

Model	Vision Encoder	LLM Backbone	Key Features
LLaVA 1.5	CLIP ViT-L	Vicuna / Mistral	Open-source; MLP projector
LLaVA-NeXT	CLIP ViT-L	Llama 3	Tiled high-res images
InternVL2	InternViT	InternLM	Top open-source as of 2024
Qwen-VL	ViT	Qwen	Strong OCR and document understanding
GPT-4V / GPT-4o	Proprietary	GPT-4	Best-in-class reasoning, natively multimodal
Gemini 1.5 Pro	Proprietary	Gemini	1M context, video/audio/image/text
Claude 3.x	Proprietary	Claude	Strong document/chart analysis

Projection Strategies

Method	Description
Linear projection	Single linear layer (LLaVA original)
MLP projector	2-layer MLP with GELU (LLaVA 1.5 — stronger)
Q-Former (BLIP-2)	Cross-attention module with learnable query tokens — compresses visual tokens
Perceiver Resampler	Fixed-size output regardless of input image resolution (Flamingo)

DALL-E 3, Imagen, and Text-to-Image at Scale

DALL-E 3 (OpenAI, 2023)

Key insight: recaptioning training data with a powerful captioner dramatically improves prompt adherence.

• Train an LLM (GPT-4V) to write dense, accurate captions for all training images
• Train the diffusion model on these high-quality synthetic captions
• At inference: GPT-4 rewrites user prompts to be more detailed before passing to the diffusion model
• Result: far better text rendering, spatial reasoning, and complex prompt adherence than DALL-E 2

Architecture

User prompt → GPT-4 prompt rewriter → Enhanced prompt
                                             ↓
                                  T5-XXL text encoder
                                             ↓
                              Cascade: 64×64 → 256×256 → 1024×1024
                                    (latent diffusion + pixel diffusion)

Imagen (Google)

• T5-XXL text encoder (11B params) instead of CLIP — captures richer text semantics
• Cascaded diffusion: generate at 64×64, then super-resolve
• Dynamic thresholding: prevents saturation at high CFG scales
• Strong on text rendering and photorealism

FLUX (Black Forest Labs, 2024)

• Diffusion Transformer (DiT) replaces U-Net entirely — attention across all spatial tokens
• Flow matching instead of DDPM noise schedule — faster convergence
• Multimodal attention: text and image tokens attend to each other in every layer
• FLUX.1[dev] is the current open-source state of the art for photorealism

GPT-4V and Gemini: Frontier Vision Capabilities

GPT-4V / GPT-4o Capabilities

Task	Capability
Visual QA	Complex reasoning about image content
OCR	Reads text from images, handwriting, PDFs
Chart/graph analysis	Extracts data, identifies trends
Code from image	Reads UI mockups, converts to code
Math from images	Solves equations photographed from textbooks
Medical imaging	Describes X-rays, MRI findings (non-diagnostic)

GPT-4o (May 2024): natively multimodal — vision, audio, and text trained together end-to-end rather than as separate modules bolted together.

Gemini 1.5 Pro

• 1 million token context window (hours of video, thousands of images)
• Natively multimodal from training: image, audio, video, text, code
• Interleaved image-text inputs: can reason across multiple images in sequence
• Strong on long-document understanding with embedded figures

Practical Evaluation Benchmarks

Benchmark	Tests
VQAv2	Visual question answering on natural images
TextVQA	Reading text in images
MMMU	College-level multimodal reasoning
DocVQA	Document understanding
ChartQA	Chart interpretation
POPE	Hallucination detection (does model invent objects?)

Summary

• CLIP: contrastive image-text pretraining creates a shared semantic space enabling zero-shot classification, retrieval, and guidance for generation
• VLMs: connect vision encoders to LLMs via projection layers — LLaVA, InternVL, GPT-4V, Gemini all follow this pattern
• DALL-E 3: recaptioning + GPT-4 prompt rewriting dramatically improves text-to-image prompt adherence
• Imagen: T5 text encoder + cascaded diffusion; FLUX: diffusion transformer with flow matching
• GPT-4o / Gemini: natively multimodal frontier models — reason across images, documents, audio, and video
• Multimodal AI is the dominant direction: every frontier model now handles multiple modalities natively

Next: LLM Alignment & RLHF — how models are made to follow instructions and behave safely.