Large Language Models

From Transformer to LLM

A Large Language Model is a decoder-only transformer trained on massive text corpora with hundreds of billions of parameters. Size alone doesn't define an LLM — the training pipeline does.

The Three-Stage Pipeline

Stage 1: Pretraining        → learns language, knowledge, reasoning
Stage 2: Instruction Tuning → learns to follow instructions
Stage 3: RLHF / DPO         → aligns with human preferences

Most deployed models (ChatGPT, Claude, Gemini) go through all three stages. Each stage shapes a different aspect of the model's behavior.

Stage 1: Pretraining

Pretraining is self-supervised learning at massive scale:

Objective: Predict the next token given all previous tokens $\mathcal{L} = -\sum_{t} \log P(x_t | x_1, ..., x_{t-1})$

Data: Hundreds of billions to trillions of tokens from web pages (CommonCrawl), books, code (GitHub), Wikipedia, scientific papers, and more.

Scale: GPT-3 used 175B parameters and 300B training tokens. LLaMA 3 70B was trained on 15 trillion tokens.

What Gets Learned?

During pretraining, the model implicitly learns:

• Grammar and syntax
• World facts and common sense
• Mathematical and logical reasoning patterns
• Code syntax and semantics
• Multilingual capabilities
• Long-range dependencies and narrative structure

This is why pretraining is the most expensive stage — but also why pretrained weights are so valuable. A pretrained base model is a general-purpose reasoning engine.

Stage 2: Instruction Tuning (SFT)

A pretrained base model completes text but doesn't follow instructions well. Supervised Fine-Tuning (SFT) teaches it the instruction-response format.

Data format: pairs of (instruction, ideal response) — typically 10K–1M examples

Instruction: Summarize the following article in 3 bullet points: [article]
Response:
• Key finding 1...
• Key finding 2...
• Key finding 3...

Data sources: Human-written demonstrations, filtered web data in instruction format, model-generated + human-filtered data (Alpaca, OpenOrca).

After SFT, the model reliably follows instructions but may still produce harmful, dishonest, or unhelpful outputs.

Stage 3: RLHF and DPO

Reinforcement Learning from Human Feedback (RLHF) aligns the model with human preferences:

1. Collect comparison data: human labelers rank model outputs (A better than B)
2. Train a Reward Model (RM): supervised learning on pairwise preferences
3. RL fine-tuning: optimize model to maximize RM score using PPO, while penalizing KL divergence from SFT model (to prevent reward hacking)

Direct Preference Optimization (DPO) — a simpler alternative:

• Skips the explicit reward model
• Directly optimizes a loss over (preferred, rejected) pairs
• Equivalent to RLHF under certain assumptions but much simpler to implement
• Used by Llama 3, Mistral, and many open-source models

Constitutional AI (Anthropic)

Claude uses a variant where a set of principles (a "constitution") guides an AI critic that critiques and revises outputs before human feedback — scaling alignment without requiring humans to label every harmful case.

import anthropic

client = anthropic.Anthropic()

# Basic completion
response = client.messages.create(
    model="claude-sonnet-4-6",
    max_tokens=1024,
    system="You are a helpful AI assistant.",
    messages=[
        {"role": "user", "content": "What are scaling laws in machine learning?"}
    ],
)
print(response.content[0].text)

# Multi-turn conversation
conversation = []
turns = [
    "What is the difference between pretraining and fine-tuning?",
    "Can you give a concrete example of each?",
    "Which one is more computationally expensive and why?",
]

for user_msg in turns:
    conversation.append({"role": "user", "content": user_msg})
    response = client.messages.create(
        model="claude-sonnet-4-6",
        max_tokens=512,
        messages=conversation,
    )
    assistant_msg = response.content[0].text
    conversation.append({"role": "assistant", "content": assistant_msg})
    print(f"User: {user_msg}")
    print(f"Claude: {assistant_msg[:150]}...")
    print()

# Token counting
token_count = client.messages.count_tokens(
    model="claude-sonnet-4-6",
    messages=[{"role": "user", "content": "How many tokens is this message?"}],
)
print(f"Token count: {token_count.input_tokens}")

Tokenization

Before feeding text to an LLM, it must be tokenized — split into sub-word pieces from a fixed vocabulary.

Byte-Pair Encoding (BPE)

The dominant approach (used by GPT, Claude, LLaMA):

1. Start with individual characters/bytes
2. Repeatedly merge the most frequent adjacent pair
3. Stop when vocabulary reaches target size (typically 32K–128K tokens)

Why Tokenization Matters

• Arithmetic is hard: "127 + 589" might be split as ["127", " +", " 589"] — each as one token — or split mid-number
• Multilingual imbalance: English words → ~1 token/word; Chinese characters → ~1.5 tokens/char
• Tokenization artifacts: capitalization, spaces, and punctuation affect token boundaries
• Context window: 1 token ≈ 4 characters in English; a 128K context ≈ ~100K words

# Using tiktoken (OpenAI's tokenizer) to inspect tokens
import tiktoken
enc = tiktoken.encoding_for_model("gpt-4")
tokens = enc.encode("Hello, world! How are you?")
print(tokens)           # [9906, 11, 1917, 0, 2650, 527, 499, 30]
print(len(tokens))      # 8

Scaling Laws

Chinchilla scaling laws (Hoffmann et al., 2022) established that for a given compute budget C:

• Optimal model size N ∝ √C
• Optimal training tokens D ∝ √C
• Rule of thumb: train on ~20 tokens per parameter

This led to smaller, better-trained models (LLaMA vs original GPT-3).

Emergent Capabilities

Some capabilities appear abruptly only beyond certain scale thresholds:

Capability	Approximate Threshold
In-context learning (few-shot)	~1B parameters
Chain-of-thought reasoning	~100B parameters
Instruction following	After RLHF (any scale)
Code generation	~10B parameters
Multi-step arithmetic	~100B parameters

Emergent abilities are controversial — some researchers argue they're measurement artifacts from metrics with discontinuous scoring.

Summary

• Pretraining on massive text teaches world knowledge and language via next-token prediction
• Instruction tuning (SFT) teaches the model to follow instructions and respond helpfully
• RLHF / DPO aligns outputs with human preferences and reduces harmful behavior
• Tokenization shapes what the model finds easy or hard
• Scaling laws guide compute allocation between model size and training tokens
• Emergent capabilities appear at scale, enabling in-context learning and reasoning

The next chapter covers Fine-Tuning & Adaptation — how to take a pretrained LLM and specialize it for your use case efficiently.