Important Hyperparameters

What Are Hyperparameters?

Hyperparameters are configuration settings that control the learning process itself — they are set before training and not learned from data.

	Parameters	Hyperparameters
Learned from?	Training data	Set by the practitioner
Examples	Weights, biases	Learning rate, depth, regularization
Stored in?	Model checkpoint	Config file

The Most Impactful Hyperparameters (80/20 rule)

Not all hyperparameters matter equally. Research consistently shows:

1. Learning rate — most impactful in any gradient-based model
2. Model capacity (depth, width, n_estimators)
3. Regularization (L2, dropout, early stopping)
4. Batch size
5. Everything else

Universal Hyperparameters

Learning Rate

The single most important hyperparameter for any gradient-based model.

$\theta \leftarrow \theta - \eta \nabla_\theta \mathcal{L}$

• Too large: loss diverges, oscillates
• Too small: very slow convergence, may get stuck
• Typical ranges: $10^{-5}$ to $10^{-1}$ (search in log scale)

Learning Rate Finder (Leslie Smith): increase LR exponentially from a tiny value; the optimal LR is just before the loss starts rising steeply.

Batch Size

• Small batches (8–32): more gradient noise → regularization effect, better generalization; slower per-epoch
• Large batches (512–4096): stable gradients, fast training; needs compensating LR increase ($\text{LR} \propto \sqrt{\text{batch\_size}}$ or linear scaling); may generalize worse
• Recommendation: 32–256 for most tasks; use largest that fits in GPU memory

Optimizer

Optimizer	Use Case
SGD + momentum	Best final performance when tuned (CNNs)
Adam	Fast convergence, good default (transformers, LSTMs)
AdamW	Adam + decoupled weight decay — best for large models
RMSProp	Good for RNNs and non-stationary objectives

Model-Specific Hyperparameters

Tree-Based Models (RF, XGBoost, LightGBM)

Parameter	Typical Range	Effect
n_estimators	100–2000	More trees = better (diminishing returns)
max_depth	3–10	Higher = more capacity, more overfitting
learning_rate (boosting)	0.01–0.3	Lower + more trees = better generalization
subsample	0.5–1.0	Row sampling per tree
colsample_bytree	0.5–1.0	Feature sampling per tree
reg_alpha (L1)	0–10	Sparsity
reg_lambda (L2)	0–10	Shrinkage
min_child_weight	1–20	Minimum sum of instance weight per leaf

Key interaction: lower learning_rate + higher n_estimators consistently improves boosted models. Start with lr=0.1, then halve it and double trees.

Neural Networks

Parameter	Typical Range	Effect
learning_rate	1e-5 to 1e-2	Most critical
hidden_size	64–2048	Capacity
n_layers	2–12	Depth
dropout	0.0–0.5	Regularization
weight_decay	1e-6 to 1e-2	L2 regularization
batch_size	16–512	Training dynamics

Tuning Strategies

Grid Search

Exhaustively try all combinations. Guaranteed to find the best in the grid, but exponential in number of parameters — impractical for more than 3–4 params.

Random Search

Sample random combinations from the search space. For the same budget, random search finds better results than grid search when some hyperparameters matter more than others.

$\text{Random search with n trials} \gg \text{Grid search with same n evaluations}$

Bayesian Optimization

Use a surrogate model (Gaussian Process or Tree-structured Parzen Estimator) to model the performance landscape and suggest the next most promising configuration — focuses evaluations where performance is likely to be high.

Optuna implements this as Tree-structured Parzen Estimator (TPE):

Method	Evaluations Needed	Parallelizable	Best For
Grid Search	$\prod n_i$	Yes	≤ 3 params
Random Search	User-defined	Yes	≤ 10 params
Bayesian (Optuna)	50–200	Partially	Any number of params

import optuna
import xgboost as xgb
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import cross_val_score, StratifiedKFold
from sklearn.metrics import roc_auc_score
import numpy as np

X, y = load_breast_cancer(return_X_y=True)
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

def objective(trial):
    """Optuna objective: return metric to maximize."""
    params = {
        "n_estimators": trial.suggest_int("n_estimators", 100, 1000),
        "max_depth": trial.suggest_int("max_depth", 3, 10),
        "learning_rate": trial.suggest_float("learning_rate", 1e-3, 0.3, log=True),
        "subsample": trial.suggest_float("subsample", 0.5, 1.0),
        "colsample_bytree": trial.suggest_float("colsample_bytree", 0.5, 1.0),
        "reg_alpha": trial.suggest_float("reg_alpha", 1e-8, 10.0, log=True),
        "reg_lambda": trial.suggest_float("reg_lambda", 1e-8, 10.0, log=True),
        "min_child_weight": trial.suggest_int("min_child_weight", 1, 20),
        "eval_metric": "logloss",
        "use_label_encoder": False,
        "random_state": 42,
    }
    model = xgb.XGBClassifier(**params)
    scores = cross_val_score(model, X, y, cv=cv, scoring="roc_auc", n_jobs=-1)
    return scores.mean()

# Run optimization (minimize by default; we set direction="maximize")
study = optuna.create_study(direction="maximize", sampler=optuna.samplers.TPESampler(seed=42))
optuna.logging.set_verbosity(optuna.logging.WARNING)
study.optimize(objective, n_trials=50, show_progress_bar=True)

print(f"Best ROC-AUC: {study.best_value:.4f}")
print(f"Best params:  {study.best_params}")

# Early stopping with XGBoost (avoid overfitting during boosting)
from sklearn.model_selection import train_test_split
X_tr, X_val, y_tr, y_val = train_test_split(X, y, test_size=0.2, stratify=y, random_state=42)
best = xgb.XGBClassifier(**study.best_params, n_estimators=5000)
best.fit(
    X_tr, y_tr,
    eval_set=[(X_val, y_val)],
    verbose=False,
    early_stopping_rounds=50,  # stop if no improvement for 50 rounds
)
print(f"Best n_estimators (early stopping): {best.best_iteration}")

Learning Rate Schedulers

After finding a good LR, use a schedule to improve convergence:

Scheduler	Behavior	Best For
StepLR	Multiply by γ every N epochs	Simple, predictable
CosineAnnealing	Smooth cosine decay to 0	General purpose, prevents getting stuck
OneCycleLR	Warm-up → peak → cool-down	Best final performance (LLMs, CNNs)
ReduceLROnPlateau	Reduce when val loss stops improving	When you don't know the epoch budget

# 1-Cycle Policy (highly recommended)
scheduler = torch.optim.lr_scheduler.OneCycleLR(
    optimizer,
    max_lr=1e-3,
    steps_per_epoch=len(train_loader),
    epochs=30,
    pct_start=0.3,     # 30% warm-up
)

Summary

• Learning rate is the most impactful hyperparameter — always tune it first, use log scale
• Model capacity (depth, width, n_estimators) and regularization (dropout, weight decay) are the next priorities
• Grid search: exhaustive but exponential — only for ≤ 3 params
• Random search: better coverage for the same budget — good general default
• Bayesian optimization (Optuna): focuses search where it matters — best for complex search spaces
• Use early stopping + LR scheduling to get the most from your training budget

Next: EDA & AutoML — exploring data and automating the full ML pipeline.