Model Evaluation Metrics & Techniques

Why Metrics Matter

Accuracy is the most intuitive metric — and often the most misleading. On a dataset where 99% of samples are negative (e.g., fraud detection), a model that always predicts "no fraud" achieves 99% accuracy while being completely useless.

Choosing the right metric is a business decision, not a technical one:

• Fraud detection: minimize false negatives (missed fraud) → maximize Recall
• Medical screening: missing a disease is worse than a false alarm → maximize Recall
• Spam filter: false positives (blocking real email) are costly → maximize Precision
• Ranking/search: care about top results → NDCG, MAP

Classification Metrics

From the confusion matrix $\begin{pmatrix} TP & FN \\ FP & TN \end{pmatrix}$:

Metric	Formula	Meaning
Accuracy	$\frac{TP+TN}{TP+TN+FP+FN}$	Overall correctness (misleading when imbalanced)
Precision	$\frac{TP}{TP+FP}$	Of all predicted positives, how many are correct?
Recall (Sensitivity)	$\frac{TP}{TP+FN}$	Of all actual positives, how many did we catch?
F1 Score	$\frac{2 \cdot P \cdot R}{P + R}$	Harmonic mean of Precision and Recall
Specificity	$\frac{TN}{TN+FP}$	True Negative Rate
F-beta	$\frac{(1+\beta^2) \cdot P \cdot R}{\beta^2 P + R}$	Weight recall $\beta$ times more than precision

Precision vs Recall Trade-off

Changing the decision threshold (default 0.5) shifts between precision and recall:

• Lower threshold → catch more positives → higher Recall, lower Precision
• Higher threshold → only confident positives → higher Precision, lower Recall

ROC Curve and AUC

The ROC curve plots True Positive Rate (Recall) vs False Positive Rate ($\frac{FP}{FP+TN}$) at all threshold values.

AUC-ROC (Area Under the Curve):

• AUC = 1.0: perfect classifier
• AUC = 0.5: random guessing (diagonal line)
• AUC = 0.0: perfect inverse classifier

Probabilistic interpretation: AUC = probability that the model ranks a random positive example higher than a random negative example.

When NOT to Use ROC-AUC

When classes are highly imbalanced, ROC-AUC is optimistic because it accounts for TN (the large, easy majority class).

Use Precision-Recall AUC instead for imbalanced problems (fraud, rare disease):

• PR curve plots Precision vs Recall
• Average Precision (AP): area under the PR curve
• No reliance on TN → more sensitive to performance on the minority class

Multi-Class Classification Metrics

For problems with $K > 2$ classes, precision/recall/F1 must be aggregated across classes:

Averaging	How	When to Use
Macro	Compute per class, then average (equal weight per class)	Balanced classes or when each class matters equally
Weighted	Compute per class, weight by class support (sample count)	Imbalanced classes — production default
Micro	Aggregate TP/FP/FN globally, then compute metric	When total instance count matters; equals accuracy for F1

$F1_{macro} = \frac{1}{K}\sum_{k=1}^{K} F1_k \qquad F1_{weighted} = \frac{\sum_{k=1}^{K} n_k \cdot F1_k}{\sum_{k=1}^{K} n_k}$

Matthews Correlation Coefficient (MCC)

The MCC is a single number that summarizes the confusion matrix even for highly imbalanced classes. It considers all four quadrants (TP, TN, FP, FN):

$MCC = \frac{TP \cdot TN - FP \cdot FN}{\sqrt{(TP+FP)(TP+FN)(TN+FP)(TN+FN)}}$

• Range: $-1$ (perfect inverse) to $+1$ (perfect) — $0$ = random
• MCC is the most informative binary classification metric for imbalanced data (even better than F1 in many cases)

Cohen's Kappa

Measures agreement between predicted and actual classes, correcting for chance agreement:

$\kappa = \frac{p_o - p_e}{1 - p_e}$

where $p_o$ = observed accuracy and $p_e$ = expected accuracy by chance. $\kappa > 0.8$ = strong agreement.

Log Loss (Cross-Entropy)

Penalizes confident wrong predictions heavily:

$\text{Log Loss} = -\frac{1}{n}\sum_{i=1}^{n}\sum_{k=1}^{K} y_{ik}\log(\hat{p}_{ik})$

Used as the training loss for probabilistic classifiers. Lower is better; 0 = perfect. Essential when predicted probabilities matter, not just the class label.

Regression Metrics

Metric	Formula	Properties
MAE	$\frac{1}{n}\sum	y_i - \hat{y}_i
MSE	$\frac{1}{n}\sum(y_i - \hat{y}_i)^2$	Penalizes large errors more, used in training loss
RMSE	$\sqrt{\text{MSE}}$	Same units as target, MSE amplified by squaring
R²	$1 - \frac{\sum(y_i - \hat{y}_i)^2}{\sum(y_i - \bar{y})^2}$	Fraction of variance explained (1=perfect, 0=baseline)
MAPE	$\frac{100}{n}\sum\left	\frac{y_i - \hat{y}_i}{y_i}\right

Rule of thumb: use MAE when you want errors in interpretable units; use RMSE when large errors are especially costly; use R² to compare models to a naive baseline.

Clustering Metrics (Unsupervised Learning)

Evaluating clustering is harder than supervised learning because there is no ground truth label to compare against. Metrics split into two families:

Internal Metrics (no ground truth needed)

These measure cluster quality using only the data and cluster assignments.

Silhouette Score

For each sample $i$, compute:

• $a(i)$ = mean distance to all other samples in the same cluster (cohesion)
• $b(i)$ = mean distance to samples in the nearest other cluster (separation)

$s(i) = \frac{b(i) - a(i)}{\max(a(i), b(i))}$

$\text{Silhouette Score} = \frac{1}{n}\sum_{i=1}^{n} s(i)$

• Range: $-1$ to $+1$
• $+1$ = sample is well inside its cluster, far from others
• $0$ = sample is near a cluster boundary
• $-1$ = sample is likely in the wrong cluster
• Best use: compare number of clusters ($k$) — pick the $k$ that maximizes silhouette score

Davies-Bouldin Index (DBI)

Average ratio of within-cluster scatter to between-cluster separation:

$DB = \frac{1}{K}\sum_{i=1}^{K} \max_{j \neq i} \frac{\sigma_i + \sigma_j}{d(c_i, c_j)}$

where $\sigma_i$ = average distance from points in cluster $i$ to centroid $c_i$, and $d(c_i, c_j)$ = distance between centroids.

• Lower is better (0 = perfect, no upper bound)
• Penalizes clusters that are spread out or too close together

Calinski-Harabasz Index (Variance Ratio Criterion)

Ratio of between-cluster dispersion to within-cluster dispersion:

$CH = \frac{\text{tr}(B_K) / (K-1)}{\text{tr}(W_K) / (n-K)}$

• Higher is better — well-separated, compact clusters score high
• Tends to favor convex, globular clusters (like K-Means)

Inertia (WCSS)

Sum of squared distances of each point to its cluster centroid:

$\text{Inertia} = \sum_{k=1}^{K}\sum_{x \in C_k} \|x - \mu_k\|^2$

• Lower is better but always decreases as $K$ increases — use the Elbow Method: plot inertia vs $K$ and pick the $K$ at the "elbow" where improvement flattens.

External Metrics (ground truth labels available)

Use these when you have true labels (e.g., to evaluate on a benchmark) but trained without them.

Metric	Description	Range	Perfect
Adjusted Rand Index (ARI)	Measures agreement between true and predicted clusters, corrected for chance	$-1$ to $1$	$1$
Normalized Mutual Information (NMI)	Mutual information normalized by cluster entropies	$0$ to $1$	$1$
Adjusted Mutual Information (AMI)	NMI corrected for chance agreement	$0$ to $1$	$1$
Fowlkes-Mallows Score	Geometric mean of pairwise precision and recall	$0$ to $1$	$1$
Homogeneity	Each cluster contains only one class	$0$ to $1$	$1$
Completeness	All members of a class are in the same cluster	$0$ to $1$	$1$
V-measure	Harmonic mean of homogeneity and completeness (like F1)	$0$ to $1$	$1$

ARI is the most commonly used external metric — it handles variable numbers of clusters and corrects for random assignments.

from sklearn.datasets import make_blobs
from sklearn.cluster import KMeans, DBSCAN
from sklearn.metrics import (
    silhouette_score, davies_bouldin_score, calinski_harabasz_score,
    adjusted_rand_score, normalized_mutual_info_score, v_measure_score,
)
import numpy as np
import matplotlib.pyplot as plt

# Synthetic data with 4 true clusters
X, y_true = make_blobs(n_samples=500, centers=4, cluster_std=0.8, random_state=42)

# ─── Elbow Method + Silhouette to find optimal K ─────────────
inertias, silhouettes, db_scores, ch_scores = [], [], [], []
K_range = range(2, 10)

for k in K_range:
    km = KMeans(n_clusters=k, random_state=42, n_init=10)
    labels = km.fit_predict(X)
    inertias.append(km.inertia_)
    silhouettes.append(silhouette_score(X, labels))
    db_scores.append(davies_bouldin_score(X, labels))
    ch_scores.append(calinski_harabasz_score(X, labels))

print(f"{'K':>4} {'Inertia':>10} {'Silhouette':>12} {'Davies-Bouldin':>16} {'Calinski-H':>12}")
print("-" * 58)
for k, inert, sil, db, ch in zip(K_range, inertias, silhouettes, db_scores, ch_scores):
    print(f"{k:>4} {inert:>10.1f} {sil:>12.4f} {db:>16.4f} {ch:>12.1f}")

# Best k: max silhouette / min Davies-Bouldin / max Calinski-H
best_k = K_range[np.argmax(silhouettes)]
print(f"\nBest K by silhouette: {best_k}")

# ─── External Metrics (when ground truth available) ──────────
km_best = KMeans(n_clusters=best_k, random_state=42, n_init=10)
y_pred = km_best.fit_predict(X)

print(f"\nExternal metrics (vs true labels):")
print(f"  Adjusted Rand Index (ARI): {adjusted_rand_score(y_true, y_pred):.4f}")
print(f"  Normalized Mutual Info:    {normalized_mutual_info_score(y_true, y_pred):.4f}")
print(f"  V-measure:                 {v_measure_score(y_true, y_pred):.4f}")

# ─── DBSCAN evaluation ───────────────────────────────────────
db = DBSCAN(eps=0.5, min_samples=5)
y_db = db.fit_predict(X)
n_clusters_db = len(set(y_db)) - (1 if -1 in y_db else 0)
n_noise = (y_db == -1).sum()
print(f"\nDBSCAN: {n_clusters_db} clusters, {n_noise} noise points")
if n_clusters_db > 1:
    # Silhouette ignores noise points (-1 labels)
    mask = y_db != -1
    print(f"  Silhouette (non-noise): {silhouette_score(X[mask], y_db[mask]):.4f}")
    print(f"  ARI:                    {adjusted_rand_score(y_true[mask], y_db[mask]):.4f}")

Ranking Metrics

Used when the model produces an ordered list of results (search, recommendations, information retrieval).

NDCG — Normalized Discounted Cumulative Gain

Measures quality of a ranked list, giving more credit to relevant items at the top:

$DCG@k = \sum_{i=1}^{k} \frac{rel_i}{\log_2(i+1)}$

$NDCG@k = \frac{DCG@k}{IDCG@k}$

where $IDCG$ = DCG of the ideal (perfect) ranking. Range: $0$ to $1$ — higher is better.

MAP — Mean Average Precision

Average of Average Precision (AP) scores across all queries:

$AP = \frac{\sum_{k=1}^{n} P(k) \cdot \text{rel}(k)}{\text{number of relevant items}}$

$MAP = \frac{1}{Q}\sum_{q=1}^{Q} AP_q$

MRR — Mean Reciprocal Rank

Average of the reciprocal rank of the first relevant item:

$MRR = \frac{1}{Q}\sum_{q=1}^{Q} \frac{1}{\text{rank}_q}$

Metric	Considers	Graded Relevance	Best For
NDCG@k	Position + relevance score	Yes	Search engines, LLM evaluation
MAP	Precision at each relevant item	Binary	IR, document retrieval
MRR	Only first relevant result	Binary	Question answering, single-answer search

Cross-Validation

Never estimate performance on the same data you trained on. Cross-validation gives an unbiased estimate of generalization performance.

K-Fold Cross-Validation

1. Split data into $k$ equal folds (typically $k=5$ or $k=10$)
2. Train on $k-1$ folds, evaluate on the held-out fold
3. Repeat $k$ times; average the scores

$\text{CV Score} = \frac{1}{k} \sum_{i=1}^{k} \text{score}_i$

Stratified K-Fold

Preserves the class distribution in each fold. Always use this for classification — especially important for imbalanced datasets.

Time-Series Cross-Validation (Walk-Forward)

For temporal data, never shuffle — use expanding or rolling windows:

Fold 1: Train [1..100]   → Test [101..110]
Fold 2: Train [1..110]   → Test [111..120]
Fold 3: Train [1..120]   → Test [121..130]

Leave-One-Out (LOO)

$k = n$ — each sample is its own validation fold. Unbiased but expensive. Only practical for very small datasets.

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import StratifiedKFold, cross_validate
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import (
    classification_report, confusion_matrix, roc_auc_score,
    average_precision_score, RocCurveDisplay, PrecisionRecallDisplay,
)
from sklearn.preprocessing import label_binarize
import matplotlib.pyplot as plt

X, y = load_breast_cancer(return_X_y=True)
model = RandomForestClassifier(n_estimators=100, random_state=42)

# ─── Cross-Validation with Multiple Metrics ─────────────────
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
results = cross_validate(
    model, X, y, cv=cv,
    scoring=["accuracy", "f1", "roc_auc", "average_precision"],
    return_train_score=True,
)

print("5-Fold CV Results:")
for metric in ["accuracy", "f1", "roc_auc", "average_precision"]:
    test_scores = results[f"test_{metric}"]
    print(f"  {metric:<22} {test_scores.mean():.4f} ± {test_scores.std():.4f}")

# ─── Detailed Evaluation on Hold-out Set ────────────────────
from sklearn.model_selection import train_test_split
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, stratify=y, random_state=42)
model.fit(X_tr, y_tr)
y_pred = model.predict(X_te)
y_proba = model.predict_proba(X_te)[:, 1]

print("
Classification Report:")
print(classification_report(y_te, y_pred, target_names=["malignant", "benign"]))

cm = confusion_matrix(y_te, y_pred)
print(f"Confusion Matrix:
{cm}")
print(f"ROC-AUC:           {roc_auc_score(y_te, y_proba):.4f}")
print(f"Average Precision: {average_precision_score(y_te, y_proba):.4f}")

# Threshold analysis
thresholds = np.arange(0.1, 1.0, 0.1)
print("
Threshold Analysis:")
print(f"{'Threshold':>12} {'Precision':>10} {'Recall':>8} {'F1':>6}")
for t in thresholds:
    from sklearn.metrics import precision_score, recall_score, f1_score
    y_t = (y_proba >= t).astype(int)
    p = precision_score(y_te, y_t, zero_division=0)
    r = recall_score(y_te, y_t)
    f = f1_score(y_te, y_t, zero_division=0)
    print(f"{t:>12.1f} {p:>10.3f} {r:>8.3f} {f:>6.3f}")

Summary

Problem	Primary Metric	Secondary
Balanced classification	Accuracy, F1 (weighted)	ROC-AUC
Imbalanced classification	PR-AUC, MCC	Recall at fixed Precision
Multi-class	F1 (weighted/macro)	Cohen's Kappa
Probabilistic classifier	Log Loss	Brier Score
Regression	RMSE or MAE	R²
Clustering (no labels)	Silhouette Score	Davies-Bouldin Index
Clustering (labels known)	Adjusted Rand Index	NMI, V-measure
Ranking / search	NDCG@k	MAP, MRR
Survival	C-index	—

• Always use stratified k-fold for classification; walk-forward for time series
• Check for data leakage — use Pipelines
• Use MCC instead of F1 for highly imbalanced binary problems
• For clustering: combine an internal metric (silhouette) with the elbow method to pick $k$
• Report confidence intervals (mean ± std across folds), not just point estimates

Next: Important Hyperparameters — a systematic guide to tuning what matters.