Semi-Supervised & Self-Supervised Learning

The Labeled Data Problem

Supervised learning requires large labeled datasets — but labeling is expensive, slow, and sometimes requires domain experts (medical images, legal documents, scientific data).

In practice, data availability looks like this:

Labeled data:    ████                (small)
Unlabeled data:  ████████████████████████████████  (large)

Semi-supervised learning leverages both labeled and unlabeled data. Self-supervised learning creates its own supervision signal from the data's structure — no human labels needed at all.

Pseudo-Labeling

The simplest semi-supervised technique:

1. Train a model on the small labeled set
2. Run the model on unlabeled data → generate pseudo-labels (predicted labels)
3. Retrain on the combined labeled + pseudo-labeled data
4. Optionally iterate (self-training loop)

Key consideration: only use high-confidence pseudo-labels (e.g., predicted probability > 0.9) to avoid propagating errors.

# Pseudo-labeling loop
model.fit(X_labeled, y_labeled)
for iteration in range(5):
    probs = model.predict_proba(X_unlabeled)
    confidence = probs.max(axis=1)
    mask = confidence > 0.95           # high-confidence only
    pseudo_labels = probs.argmax(axis=1)
    X_combined = np.vstack([X_labeled, X_unlabeled[mask]])
    y_combined = np.hstack([y_labeled, pseudo_labels[mask]])
    model.fit(X_combined, y_combined)

Label Propagation

Treats the dataset as a graph where similar points are connected. Labels "flow" from labeled to unlabeled nodes along edges.

Algorithm:

1. Build a similarity graph (e.g., $k$-NN graph, RBF kernel)
2. Labels propagate from labeled nodes to unlabeled nodes proportional to edge weight
3. Iterate until convergence

When it works best: when the manifold assumption holds — nearby points in feature space tend to have the same label. Works well for text, image embeddings.

from sklearn.semi_supervised import LabelPropagation, LabelSpreading

# -1 = unlabeled
y_semi = np.copy(y)
y_semi[unlabeled_mask] = -1

lp = LabelPropagation(kernel='knn', n_neighbors=7)
lp.fit(X, y_semi)
predicted = lp.transduction_[unlabeled_mask]

Consistency Regularization

The smoothness assumption: a model's predictions should be consistent under small perturbations of the input.

MixMatch / FixMatch

• Apply random augmentations to unlabeled images
• Generate pseudo-labels from the average of augmented predictions
• Enforce consistency: the model should output the same class for all augmented versions

$\Pi$-Model Loss

$\mathcal{L} = \mathcal{L}_{supervised} + \lambda \cdot \mathbb{E}_{x \in U}\left[\|f(x) - f(\tilde{x})\|^2\right]$

where $\tilde{x}$ is a perturbed version of $x$ and $\lambda$ is a ramp-up weight.

This family of methods (MixMatch, ReMixMatch, FixMatch) achieves remarkable results — on CIFAR-10, FixMatch with only 40 labels (4 per class!) achieves ~94% accuracy.

Self-Supervised Learning

Self-supervised learning creates a pretext task from the data itself — no human labels needed. The model learns rich representations as a byproduct.

Pretext Tasks (Historical)

• Rotation prediction: rotate an image by 0/90/180/270° and predict the angle
• Jigsaw puzzle: shuffle patches and predict the permutation
• Masked prediction: mask part of input and predict the missing part (BERT, MAE)
• Colorization: predict color from grayscale input

Contrastive Learning (Modern)

The dominant approach: learn representations where similar pairs are close and dissimilar pairs are far in embedding space.

SimCLR (Chen et al., 2020):

1. Take an image $x$, apply two random augmentations → $(x_i, x_j)$ (positive pair)
2. All other images in the batch are negatives
3. Train with NT-Xent loss (normalized temperature cross-entropy):

$\mathcal{L} = -\log \frac{\exp(\text{sim}(z_i, z_j)/\tau)}{\sum_{k \neq i} \exp(\text{sim}(z_i, z_k)/\tau)}$

Self-Supervised in NLP and Vision

BERT (NLP)

Two pretext tasks on unlabeled text:

• Masked Language Modeling (MLM): mask 15% of tokens and predict them
• Next Sentence Prediction (NSP): predict if two sentences are consecutive

Result: a pretrained encoder that can be fine-tuned on small labeled datasets for any NLP task.

GPT (NLP)

Causal Language Modeling: predict the next token. Despite being a simple objective, at scale this learns world knowledge, reasoning, and language fluency.

MAE — Masked Autoencoders (Vision)

Mask 75% of image patches; train the model to reconstruct them. Learns powerful visual representations that transfer to classification, detection, segmentation.

DINO / DINOv2 (Vision)

Self-distillation with no labels. A student network is trained to match the output of a teacher network (the teacher is an EMA of the student). Learns patch-level features useful for segmentation without any labels.

The Key Insight

Self-supervised pretraining is the reason LLMs and foundation models are so powerful — the pretraining objective forces the model to understand the data deeply, and the resulting representations transfer with minimal labels.

import numpy as np
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score

X, y = load_digits(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Simulate scarce labels: use only 5% of training data as labeled
n_labeled = int(0.05 * len(X_train))
labeled_idx = np.random.choice(len(X_train), n_labeled, replace=False)
unlabeled_idx = np.setdiff1d(np.arange(len(X_train)), labeled_idx)

X_l, y_l = X_train[labeled_idx], y_train[labeled_idx]
X_u = X_train[unlabeled_idx]

# Baseline: train only on labeled data
baseline = RandomForestClassifier(n_estimators=100, random_state=42)
baseline.fit(X_l, y_l)
print(f"Baseline (labeled only):   {accuracy_score(y_test, baseline.predict(X_test)):.3f}")

# Self-training (pseudo-labeling) loop
model = RandomForestClassifier(n_estimators=100, random_state=42)
X_combined, y_combined = X_l.copy(), y_l.copy()

for iteration in range(5):
    model.fit(X_combined, y_combined)
    probs = model.predict_proba(X_u)
    confidence = probs.max(axis=1)
    high_conf = confidence > 0.95

    if high_conf.sum() == 0:
        break

    pseudo_labels = probs.argmax(axis=1)
    X_combined = np.vstack([X_l, X_u[high_conf]])
    y_combined = np.hstack([y_l, pseudo_labels[high_conf]])

    acc = accuracy_score(y_test, model.predict(X_test))
    print(f"Iteration {iteration+1}: added {high_conf.sum():3d} pseudo-labels  acc={acc:.3f}")

print(f"Final (pseudo-labeled):    {accuracy_score(y_test, model.predict(X_test)):.3f}")

Summary

• Semi-supervised learning bridges the gap between labeled and unlabeled data
• Pseudo-labeling: train on labeled → predict unlabeled → retrain on confident predictions
• Label propagation: spread labels through a similarity graph
• Consistency regularization (FixMatch): enforce stable predictions under augmentation
• Self-supervised learning designs pretext tasks that force the model to understand data structure — no human labels needed
• The modern AI stack (BERT, GPT, MAE, CLIP) is built on self-supervised pretraining

Next: Ensemble Techniques — combining multiple models to beat any single model.