Ensemble Techniques

Why Ensembles Work: Bias-Variance Trade-off

Every model's error decomposes as:

$\text{Error} = \text{Bias}^2 + \text{Variance} + \text{Irreducible Noise}$

• Bias: error from wrong assumptions (underfitting)
• Variance: error from sensitivity to training data fluctuations (overfitting)

Model	Bias	Variance
Deep decision tree	Low	High
Shallow tree / linear model	High	Low
Ensemble of deep trees	Low	Low

Key intuition: averaging many independent predictions cancels out individual errors. Ensembles combine models to achieve low bias and low variance simultaneously.

Bagging (Bootstrap Aggregating)

Train $B$ models on bootstrap samples (random samples with replacement) of the training data, then average their predictions.

$\hat{f}_{bag}(x) = \frac{1}{B}\sum_{b=1}^{B} f_b(x)$

• Reduces variance without increasing bias
• Each model sees ~63% of data; ~37% is the out-of-bag (OOB) sample → free built-in validation set (no need to hold out data)

Random Forest — Deep Dive

Random Forest = Bagging + random feature subsets at every split

At each node split, only $m$ randomly selected features are considered (not all $p$):

• Classification: $m = \sqrt{p}$
• Regression: $m = p/3$

This decorrelates the trees — even if a strong feature dominates, not every tree can use it at every split. Diverse errors cancel out when averaged.

Why Random Forest Is Robust

1. No feature scaling needed — trees are invariant to monotone transforms
2. Handles missing values via surrogate splits (or OOB imputation)
3. Built-in feature importance — average impurity reduction (Gini/entropy) across all splits
4. OOB error — free validation estimate without a separate hold-out set
5. Parallelizable — trees are independent, training scales perfectly with cores

Hyperparameter Guide

Parameter	Default	Effect	Tuning Tip
n_estimators	100	More = better (diminishing returns after ~200)	Start at 200–500
max_features	"sqrt"	Diversity vs accuracy trade-off	"sqrt" for classification, "log2" or float for regression
max_depth	None (full)	Controls per-tree complexity	Leave None or try 10–30
min_samples_leaf	1	Minimum samples per leaf — smoothing	1–10; larger = more regularization
max_samples	1.0	Bootstrap sample size fraction	0.6–0.8 for more diversity
oob_score	False	Enable OOB validation	Set True to get free validation
class_weight	None	Weight classes inversely by frequency	Use "balanced" for imbalanced data

Feature Importance

Tree-based importance (MDI — mean decrease in impurity) is fast but can be biased toward high-cardinality features. Prefer permutation importance or SHAP for reliable rankings.

# MDI importance (fast, built-in)
rf.feature_importances_  # shape: (n_features,)

# Permutation importance (more reliable)
from sklearn.inspection import permutation_importance
result = permutation_importance(rf, X_test, y_test, n_repeats=10, random_state=42)
# result.importances_mean — average drop in score when feature is shuffled

from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.inspection import permutation_importance
from sklearn.metrics import classification_report
import numpy as np
import pandas as pd

X, y = load_breast_cancer(return_X_y=True, as_frame=True)
feature_names = X.columns.tolist()

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42
)

# ─── Train with OOB enabled ──────────────────────────────────
rf = RandomForestClassifier(
    n_estimators=300,
    max_features="sqrt",       # sqrt(30) ≈ 5 features per split
    max_depth=None,            # fully grown trees → low bias
    min_samples_leaf=2,        # slight regularization
    oob_score=True,            # free validation, no hold-out needed
    class_weight="balanced",   # handle any class imbalance
    n_jobs=-1,
    random_state=42,
)
rf.fit(X_train, y_train)

print(f"OOB accuracy:  {rf.oob_score_:.4f}")
print(f"Test accuracy: {rf.score(X_test, y_test):.4f}")
print()
print(classification_report(y_test, rf.predict(X_test),
                             target_names=["malignant", "benign"]))

# ─── MDI Feature Importance (built-in, fast) ─────────────────
mdi_imp = pd.Series(rf.feature_importances_, index=feature_names)
print("Top 10 features (MDI):")
print(mdi_imp.sort_values(ascending=False).head(10).round(4))

# ─── Permutation Importance (slower, more reliable) ──────────
perm = permutation_importance(rf, X_test, y_test, n_repeats=15, random_state=42)
perm_imp = pd.Series(perm.importances_mean, index=feature_names)
print("\nTop 10 features (Permutation Importance):")
print(perm_imp.sort_values(ascending=False).head(10).round(4))

# ─── Threshold Tuning (adjust decision boundary) ─────────────
proba = rf.predict_proba(X_test)[:, 1]
for threshold in [0.3, 0.4, 0.5, 0.6, 0.7]:
    from sklearn.metrics import f1_score, recall_score
    pred = (proba >= threshold).astype(int)
    print(f"Threshold {threshold:.1f} → Recall: {recall_score(y_test, pred):.3f}  F1: {f1_score(y_test, pred):.3f}")

Boosting

Boosting trains models sequentially, where each new model focuses on the errors of the previous ones.

AdaBoost

• Assign equal weights to all samples
• At each round: train a weak learner, increase weights of misclassified samples
• Final prediction: weighted vote of all learners

$F(x) = \text{sign}\left(\sum_{t=1}^{T} \alpha_t h_t(x)\right)$

Gradient Boosting

Frames boosting as gradient descent in function space. Each new tree fits the negative gradient (residuals) of the loss:

$F_m(x) = F_{m-1}(x) + \eta \cdot h_m(x)$

where $h_m$ is the tree fitting the residuals and $\eta$ is the learning rate (shrinkage).

Library	Speed	Key Features
XGBoost	Fast	Regularization (L1+L2), column/row subsampling, GPU
LightGBM	Fastest	Leaf-wise growth, histogram-based splits, handles 1M+ rows
CatBoost	Competitive	Native categorical handling, symmetric trees, minimal tuning

Stacking (Stacked Generalization)

Train diverse base models, then train a meta-learner on their predictions.

                   ┌────────────────────────────┐
Train data    →    │ Model 1 (RF)               │─→ predictions
                   │ Model 2 (XGBoost)          │─→ predictions  →  Meta-Learner → Final
                   │ Model 3 (Logistic Reg.)    │─→ predictions
                   └────────────────────────────┘

Critical: base models must be evaluated on out-of-fold predictions (cross-validation) to prevent data leakage when training the meta-learner.

Voting Classifier

A simpler ensemble: combine models by majority vote (hard) or average probability (soft). Soft voting almost always outperforms hard voting.

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier, StackingClassifier, VotingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.svm import SVC
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
import xgboost as xgb
import numpy as np

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Random Forest
rf = RandomForestClassifier(
    n_estimators=200,
    max_features="sqrt",
    max_depth=None,
    min_samples_leaf=2,
    n_jobs=-1,
    random_state=42,
)

# XGBoost
xgb_clf = xgb.XGBClassifier(
    n_estimators=300,
    max_depth=4,
    learning_rate=0.05,
    subsample=0.8,
    colsample_bytree=0.8,
    reg_alpha=0.1,   # L1
    reg_lambda=1.0,  # L2
    use_label_encoder=False,
    eval_metric="logloss",
    random_state=42,
)

# Stacking
base_learners = [
    ("rf", rf),
    ("xgb", xgb_clf),
    ("svm", Pipeline([("scaler", StandardScaler()), ("svm", SVC(probability=True))])),
]
stacking = StackingClassifier(
    estimators=base_learners,
    final_estimator=LogisticRegression(),
    cv=5,
    stack_method="predict_proba",
)

# Soft Voting
voting = VotingClassifier(estimators=base_learners, voting="soft")

models = {"Random Forest": rf, "XGBoost": xgb_clf,
          "Stacking": stacking, "Soft Voting": voting}

print(f"{'Model':<20} {'CV Acc':>10} {'Std':>8}")
print("-" * 42)
for name, model in models.items():
    scores = cross_val_score(model, X_train, y_train, cv=5, scoring="accuracy")
    print(f"{name:<20} {scores.mean():.4f}    ±{scores.std():.4f}")

Bagging vs Boosting vs Stacking

	Bagging	Boosting	Stacking
Training	Parallel	Sequential	Two-stage
Focus	Reduce variance	Reduce bias	Leverage diversity
Risk	High bias if trees too shallow	Overfitting if learning rate too high	Data leakage if not done with CV
Best for	High-variance models (deep trees)	Weak learners (shallow trees)	Competition, maximum accuracy
Example	Random Forest	XGBoost, LightGBM	Kaggle ensembles

Practical advice: For structured/tabular data, XGBoost or LightGBM is almost always the strongest single model. Random Forest is a strong, easy-to-tune baseline. Stacking gives a marginal boost at the cost of complexity — useful for Kaggle, overkill for production.

Summary

• Ensemble methods reduce prediction error by combining diverse models
• Bagging / Random Forest: parallel training on bootstrap samples, reduces variance
• Boosting / XGBoost / LightGBM: sequential residual fitting, reduces bias
• Stacking: two-stage — base models feed predictions to a meta-learner
• For tabular data: XGBoost/LightGBM for best performance, Random Forest for simplicity and robustness
• Feature importance and SHAP provide interpretability

Next: Deep Neural Networks — the foundation of modern AI.