Multimodal Concepts — AI Tutorials — Subhendu Datta Bhowmik

Beyond Text: The Multimodal World

The real world is inherently multimodal. Humans process text, images, audio, and video simultaneously. Modern AI systems are catching up.

Multimodal AI refers to models that understand and/or generate content across multiple modalities:

Understanding: image captioning, visual QA, document parsing
Generation: text-to-image, text-to-video, text-to-speech
Unified: models that both understand and generate across modalities

Key Modality Pairs

Input	Output	Examples
Text	Image	DALL-E 3, Stable Diffusion, Midjourney
Image	Text	Claude, GPT-4V, Gemini
Text + Image	Text	Visual QA, document AI
Text	Audio	TTS (ElevenLabs, OpenAI TTS)
Audio	Text	Whisper, Deepgram
Text	Video	Sora, Runway Gen-3

CLIP: Connecting Language and Vision

CLIP (Contrastive Language-Image Pre-Training, OpenAI 2021) is the foundational model for aligning text and image representations.

How CLIP Works

CLIP uses contrastive learning on 400M image-text pairs from the web:

Image encoder (Vision Transformer or ResNet): maps image → embedding
Text encoder (Transformer): maps caption → embedding
Training objective: maximize similarity of matching pairs, minimize for non-matching

$\mathcal{L} = -\frac{1}{N}\sum_i \log \frac{\exp(\text{sim}(I_i, T_i) / \tau)}{\sum_j \exp(\text{sim}(I_i, T_j) / \tau)}$

Why CLIP is Foundational

Image and text live in the same embedding space — enables zero-shot classification
Powers the text conditioning in Stable Diffusion, DALL-E
Zero-shot: "Is this a cat?" = compare image embedding to embeddings of "cat", "dog", etc.

Vision-Language Model with Claudepython

import anthropic
import base64
from pathlib import Path

client = anthropic.Anthropic()

def analyze_image(image_path: str, question: str) -> str:
    """Ask Claude a question about an image."""
    with open(image_path, "rb") as f:
        image_data = base64.standard_b64encode(f.read()).decode("utf-8")

    # Detect media type from extension
    ext = Path(image_path).suffix.lower()
    media_types = {".jpg": "image/jpeg", ".jpeg": "image/jpeg",
                   ".png": "image/png", ".gif": "image/gif", ".webp": "image/webp"}
    media_type = media_types.get(ext, "image/jpeg")

    response = client.messages.create(
        model="claude-sonnet-4-6",
        max_tokens=1024,
        messages=[
            {
                "role": "user",
                "content": [
                    {
                        "type": "image",
                        "source": {
                            "type": "base64",
                            "media_type": media_type,
                            "data": image_data,
                        },
                    },
                    {"type": "text", "text": question},
                ],
            }
        ],
    )
    return response.content[0].text

# Use cases
print(analyze_image("chart.png", "What trend does this chart show?"))
print(analyze_image("invoice.jpg", "Extract all line items and totals as JSON"))
print(analyze_image("ui_screenshot.png", "List all accessibility issues you can spot"))
print(analyze_image("diagram.png", "Explain this architecture diagram"))

Vision-Language Model Architectures

Modern VLMs integrate a vision encoder with an LLM:

Architecture Components

Vision Encoder: ViT (Vision Transformer) or CNN that converts image patches → visual tokens
Projection Layer: maps visual tokens to the LLM's embedding dimension
Language Model: processes the combined visual + text token sequence

Key Models

GPT-4V / GPT-4o: OpenAI's flagship multimodal model. Native multimodal training.

Claude 3.x / 4.x (Anthropic): Strong document understanding, chart analysis, and vision reasoning.

Gemini (Google): Native multimodal across text, images, audio, video. Gemini Ultra handles complex multimodal reasoning.

LLaVA / LLaVA-NeXT (open-source): CLIP vision encoder + LLaMA, training on visual instruction data.

PaliGemma (Google): Efficient open-source VLM for image captioning and VQA.

What VLMs Can Do

Document understanding: OCR, table extraction, form parsing
Chart/diagram analysis: extract data, explain trends
Visual QA: answer questions about image content
Image description: captioning, alt-text generation
Code from screenshots: "convert this UI mockup to HTML"

Text-to-Image Generation

Text-to-image models take a text prompt and generate a corresponding image. The dominant approach uses latent diffusion with CLIP-based text conditioning.

How Stable Diffusion Works

Text encoding: CLIP text encoder converts prompt → text embeddings
Latent noise: start with random Gaussian noise in a compressed latent space (not pixel space)
Denoising: U-Net iteratively denoises, conditioned on text embeddings via cross-attention
Decoding: VAE decoder maps latent → full-resolution image

Classifier-Free Guidance (CFG)

At each denoising step, blend conditional and unconditional predictions: $\hat{\epsilon} = \epsilon_{\text{uncond}} + w \cdot (\epsilon_{\text{cond}} - \epsilon_{\text{uncond}})$

where w is the guidance scale (typically 7–12). Higher w → more prompt adherent, less diverse.

Prompt Techniques for Image Generation

Style keywords: "oil painting", "photorealistic", "8k", "cinematic lighting"
Negative prompts: "blurry, distorted, low quality, watermark"
ControlNet: add structural conditioning (edges, pose, depth) for precise control
LoRA for styles: fine-tune on 10–20 images of a specific style/subject

Audio Modalities

Automatic Speech Recognition (ASR)

Whisper (OpenAI) is the dominant open-source ASR model:

Trained on 680K hours of multilingual audio
Supports 99 languages with word-level timestamps
Available via API or self-hosted

import openai
with open("audio.mp3", "rb") as f:
    transcript = openai.audio.transcriptions.create(
        model="whisper-1", file=f, response_format="verbose_json"
    )
print(transcript.text)

Text-to-Speech (TTS)

Modern neural TTS is indistinguishable from human speech:

ElevenLabs: voice cloning from 1 minute of audio, 30+ languages
OpenAI TTS: 6 voices, streamed output
Bark (open-source): speech + nonverbal sounds, multilingual

Audio Generation

MusicGen (Meta): text-to-music in any genre
AudioLDM: diffusion-based audio generation
Whisper + TTS: transcribe + translate + re-speak pipeline

Video and Beyond

Video Generation

Video generation extends image diffusion to temporal sequences:

Sora (OpenAI): text-to-video up to 60 seconds, high fidelity
Runway Gen-3: professional video generation and editing
Stable Video Diffusion: image-to-video (2–4 second clips)

Challenges in Video

Temporal consistency: objects must remain coherent across frames
Physics: motion should obey real-world dynamics
Long-form: maintaining narrative coherence over minutes
Compute: generating 1 second of 1080p video requires ~100× compute vs an image

Emerging: Omnimodal Models

Models like Gemini 1.5 and GPT-4o process audio, video, images, and text natively — not as a pipeline of specialized models but as a unified model with a single shared representation space.

Multi-Image Reasoning with Claudepython

import anthropic
import base64

client = anthropic.Anthropic()

def compare_images(image_paths: list[str], question: str) -> str:
    """Compare multiple images using Claude's vision capability."""
    content = []

    for i, path in enumerate(image_paths):
        with open(path, "rb") as f:
            data = base64.standard_b64encode(f.read()).decode("utf-8")
        ext = path.split(".")[-1].lower()
        media_type = f"image/{'jpeg' if ext in ('jpg', 'jpeg') else ext}"

        content.append({"type": "text", "text": f"Image {i+1}:"})
        content.append({
            "type": "image",
            "source": {"type": "base64", "media_type": media_type, "data": data}
        })

    content.append({"type": "text", "text": question})

    response = client.messages.create(
        model="claude-sonnet-4-6",
        max_tokens=1024,
        messages=[{"role": "user", "content": content}],
    )
    return response.content[0].text

# Compare UI designs
result = compare_images(
    ["design_v1.png", "design_v2.png"],
    "Compare these two UI designs. Which is cleaner and why?"
)
print(result)

# Analyze a data visualization
result = compare_images(
    ["sales_chart.png"],
    "Extract the monthly sales figures from this chart as JSON: {month: value}"
)
print(result)

Knowledge check

What does CLIP's contrastive learning objective accomplish?

Summary

CLIP aligns image and text in a shared embedding space via contrastive learning — the foundation of most multimodal AI
VLMs combine a vision encoder with an LLM, enabling visual understanding, document parsing, and chart analysis
Text-to-image uses latent diffusion with CLIP conditioning and classifier-free guidance
Audio AI: Whisper for ASR, neural TTS for lifelike speech, diffusion for music generation
Video generation extends diffusion temporally but faces challenges in physics and consistency
Omnimodal models unify all modalities in a single architecture

Next: Responsible AI & Governance — building AI that is safe, fair, and accountable.

← Generative AI