Supervised Learning

What Is Supervised Learning?

Supervised learning trains a model on labeled examples — pairs of input features $X$ and a known output $y$ — to learn a mapping $f: X \to y$ that generalizes to unseen data.

$\hat{y} = f(X; \theta)$

where $\theta$ are the learned parameters.

Problem Types at a Glance

Type	Output	Examples	Algorithms
Regression	Continuous value	House price, temperature	Linear, Polynomial, SVR, RF
Classification	Discrete class	Spam, disease diagnosis	Logistic, SVM, Naive Bayes, KNN
Time Series	Ordered sequence	Stock price, demand forecast	ARIMA, SARIMA, Prophet

The Supervised Learning Workflow

1. Collect labeled data $(X, y)$
2. Split into train / validation / test sets
3. Choose a model and train on the training set
4. Evaluate on the validation set; tune hyperparameters
5. Report final performance on the held-out test set

Linear Regression

Fits a linear relationship between features and a continuous output:

$\hat{y} = \theta_0 + \theta_1 x_1 + \theta_2 x_2 + \cdots + \theta_n x_n = \mathbf{\theta}^T \mathbf{x}$

Minimizes the Mean Squared Error (MSE):

$\mathcal{L} = \frac{1}{m} \sum_{i=1}^{m} (\hat{y}_i - y_i)^2$

Solved analytically via the Normal Equation: $\theta = (X^T X)^{-1} X^T y$, or iteratively via gradient descent.

Regularization

• Ridge (L2): adds $\lambda \|\theta\|_2^2$ — shrinks all coefficients, never zeros them
• Lasso (L1): adds $\lambda \|\theta\|_1$ — drives some coefficients to exactly zero (automatic feature selection)
• ElasticNet: combines L1 + L2; best when many features are correlated

Non-Linear Regression

Real-world data rarely follows a straight line. Non-linear regression extends regression to capture curves, interactions, and complex shapes.

Polynomial Regression

Add polynomial terms of existing features:

$\hat{y} = \theta_0 + \theta_1 x + \theta_2 x^2 + \theta_3 x^3 + \cdots + \theta_d x^d$

This is still linear regression in disguise — linear in the parameters $\theta$, just with engineered features $[x, x^2, x^3, \ldots]$.

• Degree $d$: controls flexibility. $d=1$ = straight line; $d=2$ = parabola; high $d$ → overfitting
• Use cross-validation to choose $d$; combine with Ridge/Lasso to regularize

Splines and Piecewise Regression

Splines divide the feature range into segments and fit a polynomial within each, smoothly joined at knot points:

• Natural cubic splines: smooth, well-behaved at boundaries
• B-splines: flexible basis functions for smooth curves

Regression with Non-linear Features (Feature Engineering)

Combine features to capture interactions and non-linearity: $\hat{y} = \theta_0 + \theta_1 x_1 + \theta_2 x_2 + \theta_3 x_1 x_2 + \theta_4 \log(x_3)$

This is still linear regression but trained on transformed/combined features.

Support Vector Regression (SVR)

SVMs applied to regression — uses the ε-insensitive tube: errors within ε of the true value incur no penalty. Automatically learns non-linear relationships via the kernel trick.

import numpy as np
from sklearn.linear_model import LinearRegression, Ridge, Lasso
from sklearn.preprocessing import PolynomialFeatures, StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.model_selection import cross_val_score
import matplotlib.pyplot as plt

# Synthetic non-linear data
np.random.seed(42)
X = np.sort(np.random.uniform(-3, 3, 100)).reshape(-1, 1)
y = 0.5 * X.ravel()**3 - X.ravel()**2 + 2 * X.ravel() + np.random.normal(0, 2, 100)

# Compare models
models = {
    "Linear":      Pipeline([("lr", LinearRegression())]),
    "Poly d=3":    Pipeline([("poly", PolynomialFeatures(3)), ("scaler", StandardScaler()), ("lr", LinearRegression())]),
    "Poly d=9":    Pipeline([("poly", PolynomialFeatures(9)), ("scaler", StandardScaler()), ("lr", LinearRegression())]),
    "Ridge d=9":   Pipeline([("poly", PolynomialFeatures(9)), ("scaler", StandardScaler()), ("ridge", Ridge(alpha=1.0))]),
    "Lasso d=9":   Pipeline([("poly", PolynomialFeatures(9)), ("scaler", StandardScaler()), ("lasso", Lasso(alpha=0.1))]),
}

print(f"{'Model':<15} {'CV R² (mean)':>14} {'± Std':>8}")
print("-" * 42)
for name, model in models.items():
    scores = cross_val_score(model, X, y, cv=5, scoring="r2")
    print(f"{name:<15} {scores.mean():>14.4f} {scores.std():>8.4f}")

# Polynomial d=3 should win (matches the true DGP)
# Polynomial d=9 overfits without regularization
# Ridge d=9 recovers via regularization

# Support Vector Regression
from sklearn.svm import SVR
svr = Pipeline([("scaler", StandardScaler()), ("svr", SVR(kernel="rbf", C=10, epsilon=0.5))])
scores = cross_val_score(svr, X, y, cv=5, scoring="r2")
print(f"{'SVR (RBF)':<15} {scores.mean():>14.4f} {scores.std():>8.4f}")

Logistic Regression

Despite the name, logistic regression is a classification algorithm. It applies the sigmoid function to a linear combination of features:

$P(y=1|x) = \sigma(\mathbf{\theta}^T \mathbf{x}) = \frac{1}{1 + e^{-\mathbf{\theta}^T \mathbf{x}}}$

Trained by minimizing Binary Cross-Entropy:

$\mathcal{L} = -\frac{1}{m}\sum_{i=1}^{m} \left[y_i \log(\hat{p}_i) + (1-y_i)\log(1-\hat{p}_i)\right]$

For multi-class problems, the softmax function generalizes logistic regression: $P(y=k|x) = \frac{e^{\theta_k^T x}}{\sum_{j} e^{\theta_j^T x}}$

When to Use

• Baseline classifier for binary/multi-class problems
• When you need calibrated probability estimates
• When interpretability (feature weights as log-odds) matters

Naive Bayes

Naive Bayes applies Bayes' theorem with the "naive" assumption that features are conditionally independent given the class:

$P(y | x_1, x_2, \ldots, x_n) \propto P(y) \prod_{i=1}^{n} P(x_i | y)$

The predicted class is the one with the highest posterior probability:

$\hat{y} = \arg\max_y P(y) \prod_{i=1}^{n} P(x_i | y)$

Variants

| Variant | Feature Type | P(x|y) Model | Use Case | |---|---|---|---| | Gaussian NB | Continuous | $\mathcal{N}(\mu_{yk}, \sigma_{yk}^2)$ | Continuous features (sensor data) | | Multinomial NB | Count data | Multinomial | Text classification (word counts) | | Bernoulli NB | Binary | Bernoulli | Binary features (word present/absent) | | Complement NB | Count data | Complement of class | Imbalanced text classification |

Why Use Naive Bayes?

• Extremely fast — closed-form training and inference
• Works well on small datasets — few parameters, low variance
• Excellent for text — Multinomial NB is a strong baseline for spam filtering, sentiment, topic classification
• Handles high-dimensional data well (no curse of dimensionality)
• Despite the "naive" assumption being almost always violated, it often works surprisingly well in practice

Laplace Smoothing

To avoid zero probabilities for unseen words: $P(x_i | y) = \frac{\text{count}(x_i, y) + \alpha}{\text{count}(y) + \alpha \cdot |V|}$ where $\alpha=1$ is Laplace smoothing and $|V|$ is the vocabulary size.

from sklearn.datasets import fetch_20newsgroups
from sklearn.feature_extraction.text import TfidfVectorizer, CountVectorizer
from sklearn.naive_bayes import MultinomialNB, GaussianNB, ComplementNB
from sklearn.pipeline import Pipeline
from sklearn.model_selection import cross_val_score
from sklearn.metrics import classification_report
import numpy as np

# ─── Text Classification ─────────────────────────────────────
categories = ['sci.space', 'rec.sport.hockey', 'talk.politics.guns', 'comp.graphics']
train = fetch_20newsgroups(subset='train', categories=categories, remove=('headers', 'footers', 'quotes'))
test  = fetch_20newsgroups(subset='test',  categories=categories, remove=('headers', 'footers', 'quotes'))

# Multinomial NB with TF-IDF — classic text classification baseline
mnb_pipeline = Pipeline([
    ("tfidf", TfidfVectorizer(max_features=10000, ngram_range=(1, 2), sublinear_tf=True)),
    ("clf",   MultinomialNB(alpha=0.1)),   # alpha = Laplace smoothing
])
mnb_pipeline.fit(train.data, train.target)
y_pred = mnb_pipeline.predict(test.data)
print("Multinomial NB — Text Classification:")
print(classification_report(test.target, y_pred, target_names=categories))

# Complement NB — better for imbalanced classes
cnb_pipeline = Pipeline([
    ("tfidf", TfidfVectorizer(max_features=10000, sublinear_tf=True)),
    ("clf",   ComplementNB(alpha=0.1)),
])
scores = cross_val_score(cnb_pipeline, train.data, train.target, cv=5, scoring="f1_macro")
print(f"Complement NB CV F1: {scores.mean():.4f} ± {scores.std():.4f}")

# ─── Gaussian NB for Continuous Features ─────────────────────
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split

X, y = load_iris(return_X_y=True)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, random_state=42)

gnb = GaussianNB()
gnb.fit(X_tr, y_tr)
print(f"
Gaussian NB (Iris): {gnb.score(X_te, y_te):.4f}")

# Inspect learned parameters: mean and variance per class per feature
print(f"Class means (shape {gnb.theta_.shape}):")
for i, cls in enumerate(gnb.classes_):
    print(f"  Class {cls}: {gnb.theta_[i].round(2)}")

Support Vector Machines (SVM)

SVMs find the maximum-margin hyperplane — the decision boundary that maximizes the distance to the nearest training examples (support vectors).

$\text{maximize} \quad \frac{2}{\|\mathbf{w}\|} \quad \text{subject to} \quad y_i(\mathbf{w}^T \mathbf{x}_i + b) \geq 1$

The Kernel Trick

SVMs implicitly map data to higher dimensions via a kernel function $K(x, z) = \phi(x)^T \phi(z)$, learning non-linear boundaries without explicitly computing the high-dimensional mapping $\phi$:

Kernel	Formula	Use Case
Linear	$K(x, z) = x^T z$	Linearly separable, high-dimensional text/NLP
RBF / Gaussian	$K(x, z) = e^{-\gamma \|x-z\|^2}$	General purpose — default choice for non-linear problems
Polynomial	$K(x, z) = (x^T z + c)^d$	Image recognition, polynomial relationships
Sigmoid	$K(x, z) = \tanh(\kappa x^T z + c)$	Neural-network-like boundaries

RBF Kernel — Deep Dive

The Radial Basis Function (RBF) kernel, also called the Gaussian kernel, is the most widely used SVM kernel. Its formula:

$K(x, z) = e^{-\gamma \|x - z\|^2}$

Intuition: $K(x, z)$ measures similarity between two points $x$ and $z$:

• When $x = z$: $\|x-z\|^2 = 0 \Rightarrow K = e^0 = 1$ (identical → maximum similarity)
• As $\|x-z\| \to \infty$: $K \to 0$ (far apart → zero similarity)

The parameter $\gamma = \frac{1}{2\sigma^2}$ controls the radius of influence of each support vector:

$\gamma$ value	Effect	Risk
Small $\gamma$ (e.g., 0.001)	Large radius — each support vector influences a wide area → smooth, global boundary	Underfitting (high bias)
Large $\gamma$ (e.g., 100)	Small radius — each support vector influences only nearby points → complex, wiggly boundary	Overfitting (high variance)

Interaction: C and γ

C and γ must be tuned together — they jointly control the decision boundary:

	Small C	Large C
Small γ	Very smooth, possibly under-fit	Smoother boundary, few support vectors
Large γ	Localized, complex	Very tight fit, many support vectors, high overfit risk

Best practice: search both on a log scale — try $C in [0.01, 1, 10, 100, 1000]$ and $\gamma in [10^{-4}, 10^{-3}, 10^{-2}, 0.1, 1]$ using cross-validation. Set gamma="scale" (default in sklearn) to auto-set $\gamma = \frac{1}{p \cdot \text{Var}(X)}$.

When to Use Which Kernel

• Linear: $n < d$ (more features than samples), text classification, sparse data
• RBF: default for tabular data with non-linear structure; start here when in doubt
• Polynomial: when domain knowledge suggests polynomial interactions (images)
• Sigmoid: rarely used — neural networks usually outperform it

K-Nearest Neighbors (KNN)

KNN is a lazy learner — it memorizes all training examples and classifies by majority vote (classification) or average (regression) among the $k$ nearest neighbors at prediction time.

$\hat{y} = \frac{1}{k} \sum_{i \in \mathcal{N}_k(x)} y_i \quad \text{(regression)} \qquad \hat{y} = \text{majority}_{i \in \mathcal{N}_k(x)}(y_i) \quad \text{(classification)}$

Distance metrics: Euclidean ($L_2$), Manhattan ($L_1$), Minkowski, Cosine (for text embeddings)

Pros and Cons

Pros	Cons
No training time	Slow prediction: $O(nd)$ per query
Naturally multi-class	Sensitive to irrelevant/scaled features
Captures non-linear boundaries	Poor in high dimensions (curse of dimensionality)
No assumptions about data distribution	Large memory footprint

Rule of thumb for k: start with $k = \sqrt{n}$; always tune via cross-validation. Odd $k$ avoids ties in binary classification.

Decision Trees

Decision trees recursively split the data on feature thresholds to minimize impurity:

• Gini impurity: $G = 1 - \sum_k p_k^2$ (faster to compute)
• Entropy / Information Gain: $H = -\sum_k p_k \log_2 p_k$ (theoretically cleaner)

At each node, the algorithm picks the feature and threshold that maximize information gain (largest impurity reduction).

Key Hyperparameters

Parameter	Effect
max_depth	Limits tree depth — primary lever against overfitting
min_samples_split	Minimum samples required to split a node
min_samples_leaf	Minimum samples in any leaf — smoothing
criterion	'gini' or 'entropy'
max_features	Features considered per split

Pros and Cons

• Pros: interpretable (can visualize), handles mixed types, no feature scaling needed, captures non-linear relationships
• Cons: high variance (unstable — small changes → very different tree), prone to overfitting without pruning

Decision trees alone are rarely best — they shine as base learners in Random Forests and Gradient Boosting (covered in the Ensembles chapter).

Random Forest (Preview)

Random Forest = many decision trees trained on bootstrap samples + random feature subsets. The averaging of diverse trees dramatically reduces variance. It is one of the most powerful and robust algorithms for tabular data — see Chapter 5 for a deep dive.

Time Series Forecasting: ARIMA & SARIMA

Time series data has a temporal ordering — future values depend on past values. Standard ML models that assume i.i.d. data don't apply directly. Classical statistical approaches are often highly effective.

Stationarity

A time series is stationary if its mean and variance are constant over time. ARIMA requires stationarity.

Test for stationarity: Augmented Dickey-Fuller (ADF) test

• $H_0$: series has a unit root (non-stationary)
• If $p < 0.05$: reject $H_0$ → series is stationary

Make stationary: differencing — $y'_t = y_t - y_{t-1}$, or log transform for exponential trends.

---

ARIMA (AutoRegressive Integrated Moving Average)

ARIMA$(p, d, q)$ combines three components:

$\underbrace{\phi(B)}_{\text{AR}(p)} \underbrace{(1-B)^d}_{\text{I}(d)} y_t = \underbrace{\theta(B)}_{\text{MA}(q)} \epsilon_t$

Parameter	Component	Meaning
$p$	AR — AutoRegressive	Depends on previous $p$ values: $\sum_{i=1}^p \phi_i y_{t-i}$
$d$	I — Integrated	Apply differencing $d$ times to achieve stationarity
$q$	MA — Moving Average	Depends on previous $q$ forecast errors: $\sum_{j=1}^q \theta_j \epsilon_{t-j}$

Choosing p, d, q:

• d: number of differences needed for stationarity (ADF test)
• p: from PACF (Partial AutoCorrelation Function) — cut-off lag
• q: from ACF (AutoCorrelation Function) — cut-off lag
• Or use auto_arima to search automatically

---

SARIMA (Seasonal ARIMA)

SARIMA$(p, d, q)(P, D, Q)_m$ adds seasonal components for periodic patterns (daily, weekly, yearly):

$\Phi_P(B^m) \phi_p(B) (1-B^m)^D (1-B)^d y_t = \Theta_Q(B^m) \theta_q(B) \epsilon_t$

Parameter	Meaning
$P, D, Q$	Seasonal AR, differencing, MA orders
$m$	Seasonal period (12 = monthly with yearly seasonality; 7 = daily with weekly)

When to use SARIMA over ARIMA: when ACF shows significant spikes at multiples of $m$ (seasonal lags).

import pandas as pd
import numpy as np
from statsmodels.tsa.statespace.sarimax import SARIMAX
from statsmodels.tsa.stattools import adfuller, acf, pacf
from statsmodels.graphics.tsaplots import plot_acf, plot_pacf
from sklearn.metrics import mean_absolute_error, mean_squared_error
import warnings
warnings.filterwarnings("ignore")

# ─── Generate sample time series ─────────────────────────────
np.random.seed(42)
n = 200
t = np.arange(n)
trend = 0.05 * t
seasonality = 5 * np.sin(2 * np.pi * t / 12)   # monthly seasonality
noise = np.random.normal(0, 1, n)
y = pd.Series(50 + trend + seasonality + noise,
              index=pd.date_range("2007-01", periods=n, freq="ME"))

# ─── Stationarity Check ───────────────────────────────────────
adf_result = adfuller(y, autolag="AIC")
print(f"ADF p-value: {adf_result[1]:.4f} ({'stationary' if adf_result[1] < 0.05 else 'non-stationary'})")

# First difference (d=1) usually achieves stationarity for trended data
y_diff = y.diff().dropna()
adf_diff = adfuller(y_diff, autolag="AIC")
print(f"ADF after differencing: {adf_diff[1]:.4f} ({'stationary' if adf_diff[1] < 0.05 else 'non-stationary'})")

# ─── Train/Test Split ────────────────────────────────────────
train, test = y[:-24], y[-24:]

# ─── ARIMA (no seasonal component) ───────────────────────────
arima = SARIMAX(train, order=(2, 1, 2), trend="n")
arima_fit = arima.fit(disp=False)
arima_forecast = arima_fit.forecast(steps=24)
arima_mae = mean_absolute_error(test, arima_forecast)
print(f"
ARIMA(2,1,2)  MAE: {arima_mae:.3f}")

# ─── SARIMA (with yearly seasonality m=12) ───────────────────
sarima = SARIMAX(
    train,
    order=(1, 1, 1),          # non-seasonal: p,d,q
    seasonal_order=(1, 1, 1, 12),  # seasonal: P,D,Q,m
    trend="n",
)
sarima_fit = sarima.fit(disp=False)
sarima_forecast = sarima_fit.forecast(steps=24)
sarima_mae = mean_absolute_error(test, sarima_forecast)
print(f"SARIMA(1,1,1)(1,1,1)12 MAE: {sarima_mae:.3f}")

print(f"
Model summary:")
print(sarima_fit.summary().tables[0])

# ─── Auto ARIMA (finds best p,d,q automatically) ─────────────
# pip install pmdarima
from pmdarima import auto_arima
auto_model = auto_arima(
    train,
    seasonal=True, m=12,
    stepwise=True,            # faster than exhaustive search
    information_criterion="aic",
    suppress_warnings=True,
)
print(f"
auto_arima selected: {auto_model.order} seasonal={auto_model.seasonal_order}")
auto_forecast = auto_model.predict(n_periods=24)
print(f"auto_arima MAE: {mean_absolute_error(test, auto_forecast):.3f}")

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.linear_model import LogisticRegression
from sklearn.svm import SVC
from sklearn.tree import DecisionTreeClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.naive_bayes import GaussianNB
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import classification_report
import numpy as np

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

models = {
    "Logistic Regression":  Pipeline([("sc", StandardScaler()), ("clf", LogisticRegression(C=1.0, max_iter=1000))]),
    "SVM (RBF kernel)":     Pipeline([("sc", StandardScaler()), ("clf", SVC(C=1.0, kernel="rbf", gamma="scale"))]),
    "Decision Tree":        Pipeline([("clf", DecisionTreeClassifier(max_depth=5, min_samples_leaf=5))]),
    "KNN (k=7)":            Pipeline([("sc", StandardScaler()), ("clf", KNeighborsClassifier(n_neighbors=7))]),
    "Gaussian Naive Bayes": Pipeline([("clf", GaussianNB())]),
    "Random Forest":        Pipeline([("clf", RandomForestClassifier(n_estimators=100, random_state=42))]),
}

print(f"{'Model':<28} {'CV Accuracy':>12} {'Std':>8}")
print("-" * 52)
for name, pipeline in models.items():
    scores = cross_val_score(pipeline, X_train, y_train, cv=5, scoring="accuracy")
    print(f"{name:<28} {scores.mean():.4f}      ±{scores.std():.4f}")

# Best model final evaluation
best = models["SVM (RBF kernel)"]
best.fit(X_train, y_train)
print("\nTest set — SVM (RBF):")
print(classification_report(y_test, best.predict(X_test), target_names=["malignant", "benign"]))

Full Algorithm Summary

Regression Algorithms

Algorithm	Best For	Key Weakness
Linear Regression	Fast baseline, interpretable	Linear boundaries only
Polynomial Regression	Smooth non-linear curves	Overfits at high degree
Ridge / Lasso	Many correlated features	Still linear/polynomial
SVR	Small-medium datasets, non-linear	Slow on large data
Random Forest Regressor	Tabular data, non-linear	Black box, memory

Classification Algorithms

Algorithm	Best For	Key Weakness
Logistic Regression	Baseline, probabilities, interpretable	Linear boundary
Naive Bayes	Text, small data, fast	Independence assumption
SVM	High-dim, small-medium datasets	Slow $O(n^2)$–$O(n^3)$ training
KNN	Simple, no training needed	Slow inference $O(nd)$
Decision Tree	Interpretability	High variance
Random Forest	Tabular data — default first choice	See Chapter 5

Time Series Algorithms

Algorithm	Best For	Key Weakness
ARIMA$(p,d,q)$	Stationary or differenced series	No seasonality handling
SARIMA$(p,d,q)(P,D,Q)_m$	Series with periodic seasonality	Many parameters to tune
Prophet	Business time series, holidays	Less flexible for complex patterns

Next: Unsupervised Learning — discovering patterns in unlabeled data.