Fine-Tuning & Adaptation

Should You Fine-Tune?

Fine-tuning is often the wrong first step. Before committing to training, consider:

Approach	Best When	Cost
Prompting	General tasks, quick iteration	Zero
RAG	Need up-to-date or private knowledge	Low–medium
Fine-tuning	Specific style/format, task mastery, efficiency	High
Pretraining from scratch	Unique domain language, full control	Very high

Fine-tune when:

• You need consistent output format (JSON schemas, structured reports)
• Task requires deep domain expertise the model lacks
• You want faster/cheaper inference (smaller fine-tuned model > larger prompted model)
• The task isn't well-solved by prompting after 20+ iterations

Full Fine-Tuning

Full fine-tuning updates all model parameters on task-specific data:

• Pros: Maximum flexibility, best performance ceiling
• Cons: Requires as much GPU memory as pretraining; risk of catastrophic forgetting (overwriting general capabilities); separate copy per task

For a 7B parameter model in fp16, full fine-tuning requires ~112GB VRAM (model weights + optimizer states + gradients). This means 4+ A100s for even the smallest modern LLMs.

Training Data Format

Most fine-tuning uses a chat template that mirrors the instruction-tuning format:

{
  "messages": [
    {"role": "system", "content": "You are a medical coding assistant."},
    {"role": "user", "content": "Code this diagnosis: Patient has type 2 diabetes with nephropathy"},
    {"role": "assistant", "content": "ICD-10: E11.65 (Type 2 diabetes mellitus with hyperglycemia) + N18.9 (Chronic kidney disease, unspecified)"}
  ]
}

LoRA: Low-Rank Adaptation

LoRA (Hu et al., 2021) is the most widely used PEFT technique. The key insight: weight updates during fine-tuning have low intrinsic rank.

Instead of learning ΔW (d×d), LoRA decomposes it: $\Delta W = BA$

where B ∈ ℝ^{d×r} and A ∈ ℝ^{r×d}, with rank r << d.

At inference: W' = W + BA (or merged for zero overhead)

Why This Works

• A 4096×4096 weight matrix has 16M parameters
• With r=16: B (4096×16) + A (16×4096) = only 131K parameters — 99.2% reduction
• Only A and B are trained; the original W is frozen
• Multiple LoRA adapters can be swapped in/out for different tasks

Typical Hyperparameters

• Rank (r): 4–64 (higher = more capacity, more parameters)
• Alpha (α): scaling factor, usually 2r or r
• Target modules: typically query and value projections (q_proj, v_proj)

from transformers import AutoModelForCausalLM, AutoTokenizer, TrainingArguments
from peft import LoraConfig, get_peft_model, TaskType
from trl import SFTTrainer
import torch

# Load base model
model_name = "meta-llama/Llama-3.2-3B-Instruct"
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained(model_name)
tokenizer.pad_token = tokenizer.eos_token

# Configure LoRA
lora_config = LoraConfig(
    task_type=TaskType.CAUSAL_LM,
    r=16,               # rank
    lora_alpha=32,      # scaling = alpha / r
    target_modules=["q_proj", "v_proj", "k_proj", "o_proj"],
    lora_dropout=0.05,
    bias="none",
)

# Wrap model with LoRA
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# trainable params: 6,815,744 || all params: 3,219,816,448 || trainable%: 0.2117

# Training
training_args = TrainingArguments(
    output_dir="./lora-output",
    num_train_epochs=3,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    learning_rate=2e-4,
    fp16=True,
    logging_steps=10,
    save_strategy="epoch",
    warmup_ratio=0.03,
)

trainer = SFTTrainer(
    model=model,
    args=training_args,
    train_dataset=your_dataset,   # replace with your dataset
    dataset_text_field="text",
    max_seq_length=2048,
)
trainer.train()

# Save LoRA adapter (only ~20MB, not 6GB)
model.save_pretrained("./lora-adapter")

QLoRA: Quantized LoRA

QLoRA (Dettmers et al., 2023) combines LoRA with 4-bit quantization to fine-tune 65B models on a single 48GB GPU:

1. 4-bit NormalFloat (NF4): quantize the frozen base model weights to 4 bits
2. Double quantization: quantize the quantization constants themselves
3. Paged optimizers: use CPU memory for optimizer states during memory spikes
4. Train LoRA adapters in bfloat16 on top of the quantized backbone

Memory comparison for a 7B model:

Method	VRAM
Full FT (fp16)	~112 GB
LoRA (fp16)	~16 GB
QLoRA (4-bit)	~6 GB

This democratized fine-tuning — a single consumer GPU (RTX 3090/4090) can now fine-tune 7B models.

from transformers import AutoModelForCausalLM, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
import torch

# 4-bit quantization config
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=True,    # double quantization
    bnb_4bit_quant_type="nf4",         # NormalFloat4
    bnb_4bit_compute_dtype=torch.bfloat16,
)

# Load model in 4-bit
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.2-7B-Instruct",
    quantization_config=bnb_config,
    device_map="auto",
)

# Prepare for k-bit training (handles gradient checkpointing, etc.)
model = prepare_model_for_kbit_training(model)

# Add LoRA on top
lora_config = LoraConfig(
    r=64,
    lora_alpha=16,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                    "gate_proj", "up_proj", "down_proj"],  # include FFN for more capacity
    lora_dropout=0.1,
    bias="none",
    task_type="CAUSAL_LM",
)
model = get_peft_model(model, lora_config)

# Now train normally — uses only ~6GB VRAM for a 7B model
print(f"GPU Memory: {torch.cuda.memory_allocated() / 1e9:.2f} GB")

Other PEFT Techniques

Adapters

Insert small trainable modules (linear layers) between frozen transformer layers. Slightly more overhead than LoRA at inference unless merged.

Prefix Tuning / P-Tuning

Learn soft prompt vectors prepended to each layer's key-value pairs. No modification to model architecture. Useful when you can't modify weights.

(IA)³ (Infused Adapter by Inhibiting and Amplifying Inner Activations)

Scales activations with learned vectors — even fewer parameters than LoRA (only ~0.01% of parameters trainable).

When to Use Each

Method	Parameters	Memory	Flexibility	Best For
Full FT	100%	Very high	Maximum	Unlimited resources
LoRA	0.1–1%	Low	High	Most use cases
QLoRA	0.1–1%	Very low	High	Consumer GPUs
Prefix Tuning	<0.1%	Minimal	Limited	Frozen model APIs

Summary

• Fine-tune when prompting and RAG aren't sufficient, especially for format consistency and domain mastery
• Full fine-tuning offers maximum flexibility but requires significant GPU resources
• LoRA decomposes weight updates into low-rank matrices, reducing trainable parameters by ~99%
• QLoRA adds 4-bit quantization to LoRA, enabling 7B+ model fine-tuning on consumer GPUs
• Data quality is more important than quantity — curate carefully
• After fine-tuning, you can merge LoRA weights into the base model for zero inference overhead

Next, we'll cover Prompt Engineering — how to get the most out of LLMs without any training.