EDA & AutoML — AI Tutorials — Subhendu Datta Bhowmik

What Is EDA?

Exploratory Data Analysis (EDA) is the process of understanding your data before modeling — discovering patterns, detecting anomalies, testing assumptions, and generating hypotheses.

Introduced by John Tukey (1977), EDA is non-negotiable: the best model on bad/misunderstood data will perform poorly. EDA prevents:

Training on the wrong target (target leakage)
Feeding data in the wrong scale or distribution
Missing the most important features
Ignoring class imbalance

EDA Checklist

1. Shape and dtypes
2. Missing values (count, pattern, mechanism)
3. Target distribution
4. Univariate distributions (each feature)
5. Bivariate analysis (feature vs target)
6. Correlations
7. Outliers
8. Temporal patterns (if time series)
9. Feature interactions

Structured EDA Pipelinepython

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from scipy import stats

df = pd.read_csv("your_dataset.csv")

# ─── 1. Basic Overview ───────────────────────────────────────
print("Shape:", df.shape)
print("
Dtypes:
", df.dtypes.value_counts())
print("
Head:
", df.head(3))
print("
Describe:
", df.describe())

# ─── 2. Missing Values ──────────────────────────────────────
missing = df.isnull().sum()
missing_pct = (missing / len(df) * 100).sort_values(ascending=False)
print("
Missing values (%):
", missing_pct[missing_pct > 0])

# Missing value heatmap (pattern detection)
# sns.heatmap(df.isnull(), yticklabels=False, cbar=False, cmap="viridis")

# ─── 3. Target Distribution ─────────────────────────────────
target = "price"   # or your target column
print(f"
Target ({target}):")
print(f"  Skewness: {df[target].skew():.3f}")
print(f"  Kurtosis: {df[target].kurtosis():.3f}")
# Skew > 1 → consider log transform

# ─── 4. Numeric Feature Distributions ───────────────────────
numeric_cols = df.select_dtypes(include=np.number).columns.tolist()
fig, axes = plt.subplots(len(numeric_cols)//3 + 1, 3, figsize=(15, 20))
for ax, col in zip(axes.flatten(), numeric_cols):
    df[col].hist(bins=50, ax=ax)
    ax.set_title(col)
    ax.set_xlabel("")
plt.tight_layout()

# ─── 5. Correlation Matrix ──────────────────────────────────
corr = df[numeric_cols].corr()
plt.figure(figsize=(12, 10))
mask = np.triu(np.ones_like(corr, dtype=bool))
sns.heatmap(corr, mask=mask, annot=True, fmt=".2f",
            cmap="coolwarm", center=0, square=True)
plt.title("Correlation Matrix")

# High correlations with target
print("
Top correlations with target:")
print(corr[target].abs().sort_values(ascending=False).head(10))

# ─── 6. Outlier Detection ───────────────────────────────────
for col in numeric_cols[:5]:
    z_scores = np.abs(stats.zscore(df[col].dropna()))
    outliers = (z_scores > 3).sum()
    iqr = df[col].quantile(0.75) - df[col].quantile(0.25)
    iqr_outliers = ((df[col] < df[col].quantile(0.25) - 1.5 * iqr) |
                    (df[col] > df[col].quantile(0.75) + 1.5 * iqr)).sum()
    print(f"{col:<20} Z-score outliers: {outliers:4d}  IQR outliers: {iqr_outliers:4d}")

Feature Engineering

Feature engineering transforms raw data into informative features that ML models can learn from. It is often the highest-leverage activity in an ML project.

Numeric Features

Log transform: reduces skewness — apply when feature is right-skewed ( $x \leftarrow \log(1+x)$ )
Polynomial features: $x_1^2, x_1 x_2$ — capture non-linear relationships
Binning: convert continuous to ordinal (age groups, price ranges)
Scaling: StandardScaler, MinMaxScaler, RobustScaler (for outliers)

Categorical Features

One-Hot Encoding: $k$ binary columns; never use for high-cardinality (>20 categories)
Label Encoding: ordinal for tree models; don't use for linear models
Target Encoding: replace category with mean of target — powerful but leakage risk; use with CV
Frequency Encoding: replace category with count/frequency — good for high-cardinality

Datetime Features

df["hour"] = df["timestamp"].dt.hour
df["day_of_week"] = df["timestamp"].dt.dayofweek
df["is_weekend"] = (df["day_of_week"] >= 5).astype(int)
df["month"] = df["timestamp"].dt.month
df["days_since_event"] = (df["timestamp"] - reference_date).dt.days

Text Features

TF-IDF: term frequency × inverse document frequency
Count Vectorizer: raw word counts
Embeddings: sentence-transformers, TF-IDF + SVD (LSA)

Dimensionality Reduction

High-dimensional data causes the curse of dimensionality — distance metrics become meaningless, models overfit, and training slows down. Dimensionality reduction projects data from $p$ dimensions to $k ll p$ dimensions while preserving the most important structure.

When to Apply Dimensionality Reduction

Situation	Use Case
Too many features for a linear model	PCA before Ridge/Logistic Regression
Visualization of high-dimensional clusters	UMAP / t-SNE to 2D
Noisy/redundant features (text TF-IDF, image pixels)	PCA / Truncated SVD
Class-discriminative reduction	LDA (supervised)
Speeding up downstream models	Reduce features before expensive training

PCA — Principal Component Analysis

PCA finds the directions (principal components) of maximum variance in the data and projects onto them:

$Z = XW$

where $W in \mathbb{R}^{p \times k}$ contains the top- $k$ eigenvectors of the covariance matrix $\Sigma = \frac{1}{n}X^TX$ .

Key properties:

Components are orthogonal — no correlation between them
First component captures the most variance; each subsequent captures less
Explained variance ratio: $\frac{\lambda_i}{\sum_j \lambda_j}$ — choose $k$ so cumulative explained variance ≥ 95%
Requires scaling — always StandardScale before PCA

$\text{Choose } k \text{ such that } \sum_{i=1}^{k} \frac{\lambda_i}{\sum_j \lambda_j} \geq 0.95$

LDA — Linear Discriminant Analysis (Supervised)

LDA maximizes between-class scatter while minimizing within-class scatter — unlike PCA which ignores class labels:

$J(W) = \frac{W^T S_B W}{W^T S_W W}$

where $S_B$ = between-class scatter matrix, $S_W$ = within-class scatter matrix.

Maximum $k = C - 1$ components (where $C$ = number of classes)
Better than PCA when the goal is classification — finds class-discriminative directions
Assumes Gaussian class distributions with equal covariance

t-SNE vs UMAP (Non-linear, for Visualization)

	t-SNE	UMAP
Algorithm	Minimizes KL divergence between neighbor distributions	Preserves topological structure via fuzzy simplicial sets
Speed	Slow — $O(n^2)$ or $O(n \log n)$ with Barnes-Hut	Much faster — $O(n^{1.14})$
Global structure	Often lost — clusters can be misleading	Better preserved
Stability	Different runs give different layouts	More stable
Use for ML features	No — stochastic, non-invertible	Yes — can transform new data
Best for	Visualizing cluster structure	Visualization + as ML features

Critical note: t-SNE and UMAP are for visualization only unless UMAP is explicitly used as a feature extractor. Never use t-SNE/UMAP components as input to a classifier and report results — the stochastic nature makes it invalid.

Sparse / Text: Truncated SVD (LSA)

For sparse matrices (TF-IDF, document-term), standard PCA is impractical. Truncated SVD (Latent Semantic Analysis) applies directly to sparse matrices:

$X \approx U \Sigma V^T \quad \text{(top-}k\text{ singular values)}$

from sklearn.decomposition import TruncatedSVD
from sklearn.feature_extraction.text import TfidfVectorizer

tfidf = TfidfVectorizer(max_features=10000)
X_sparse = tfidf.fit_transform(documents)        # sparse: (n_docs, 10000)
svd = TruncatedSVD(n_components=100, random_state=42)
X_dense = svd.fit_transform(X_sparse)            # dense: (n_docs, 100)
print(f"Explained variance: {svd.explained_variance_ratio_.sum():.3f}")

PCA, LDA, and UMAP — Practical Dimensionality Reductionpython

import numpy as np
from sklearn.datasets import load_digits
from sklearn.decomposition import PCA
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score

X, y = load_digits(return_X_y=True)   # 1797 samples, 64 features (8x8 images), 10 classes

# ─── PCA: Choose k by explained variance ─────────────────────
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

pca_full = PCA().fit(X_scaled)
cumvar = np.cumsum(pca_full.explained_variance_ratio_)
k_95 = np.searchsorted(cumvar, 0.95) + 1
k_99 = np.searchsorted(cumvar, 0.99) + 1
print(f"Components for 95% variance: {k_95} (out of 64)")
print(f"Components for 99% variance: {k_99}")

# ─── Compare classification accuracy ─────────────────────────
configs = {
    "Original (64 features)":    Pipeline([("sc", StandardScaler()), ("clf", LogisticRegression(max_iter=1000))]),
    f"PCA-{k_95} (95% var)":     Pipeline([("sc", StandardScaler()), ("pca", PCA(n_components=k_95)), ("clf", LogisticRegression(max_iter=1000))]),
    "PCA-20 (aggressive)":       Pipeline([("sc", StandardScaler()), ("pca", PCA(n_components=20)),   ("clf", LogisticRegression(max_iter=1000))]),
    "PCA-10 (very aggressive)":  Pipeline([("sc", StandardScaler()), ("pca", PCA(n_components=10)),   ("clf", LogisticRegression(max_iter=1000))]),
    f"LDA-{min(9, k_95)} (supervised)": Pipeline([("sc", StandardScaler()), ("lda", LinearDiscriminantAnalysis(n_components=9)), ("clf", LogisticRegression(max_iter=1000))]),
}

print(f"\n{'Config':<35} {'CV Accuracy':>12}")
print("-" * 50)
for name, pipe in configs.items():
    scores = cross_val_score(pipe, X, y, cv=5, scoring="accuracy")
    print(f"{name:<35} {scores.mean():.4f} ± {scores.std():.4f}")

# ─── UMAP for visualization (and features) ───────────────────
# pip install umap-learn
try:
    import umap
    reducer = umap.UMAP(n_components=2, n_neighbors=15, min_dist=0.1, random_state=42)
    X_umap_2d = reducer.fit_transform(X_scaled)
    print(f"\nUMAP 2D shape: {X_umap_2d.shape}")

    # UMAP as ML features (using more components)
    reducer_feat = umap.UMAP(n_components=20, n_neighbors=15, random_state=42)
    X_umap_feat = reducer_feat.fit_transform(X_scaled)
    pipe_umap = LogisticRegression(max_iter=1000)
    scores_umap = cross_val_score(pipe_umap, X_umap_feat, y, cv=5, scoring="accuracy")
    print(f"UMAP-20 features CV accuracy: {scores_umap.mean():.4f} ± {scores_umap.std():.4f}")
except ImportError:
    print("Install umap-learn: pip install umap-learn")

# ─── Reconstruction error (PCA quality check) ────────────────
pca_k = PCA(n_components=k_95)
X_reduced = pca_k.fit_transform(X_scaled)
X_reconstructed = pca_k.inverse_transform(X_reduced)
reconstruction_error = np.mean((X_scaled - X_reconstructed) ** 2)
print(f"\nPCA-{k_95} reconstruction MSE: {reconstruction_error:.4f}")

Missing Value Handling

Strategy	When to Use
Drop rows	<1% missing, random mechanism (MCAR)
Drop columns	>50% missing, not important
Mean/median imputation	Numeric, small % missing, random
Mode imputation	Categorical, small % missing
KNN imputation	Complex missing patterns
Iterative imputation (MICE)	Missing not at random (MNAR/MAR)
Add missing indicator	Always add when imputing — signal itself may matter

Missing Data Mechanisms

MCAR (Missing Completely At Random): no pattern — safest to impute or drop
MAR (Missing At Random): depends on observed values — use model-based imputation
MNAR (Missing Not At Random): depends on the missing value itself — hardest; may need domain knowledge

AutoML

AutoML automates the repetitive, tedious parts of the ML pipeline: preprocessing, feature engineering, model selection, and hyperparameter tuning.

What AutoML Automates

Raw Data
  ↓
[AUTO] Data preprocessing (imputation, encoding, scaling)
  ↓
[AUTO] Feature engineering (interactions, transforms)
  ↓
[AUTO] Model selection (RF, XGB, NNs, linear models, ...)
  ↓
[AUTO] Hyperparameter optimization (Bayesian, TPE, ...)
  ↓
[AUTO] Ensembling (stack best models)
  ↓
Final Model

Popular AutoML Tools

Tool	Backend	Best For
AutoSklearn	scikit-learn	Tabular data, reproducible research
H2O AutoML	Java/Python	Large datasets, production, interpretability
TPOT	Genetic programming	Exploring novel pipelines
AutoGluon	PyTorch/XGB	Tabular + images + text, fast & strong
Vertex AI AutoML	Google Cloud	Managed cloud, no-code
Azure AutoML	Azure	Enterprise, MLOps integration

AutoML with AutoGluonpython

from autogluon.tabular import TabularDataset, TabularPredictor
from sklearn.datasets import fetch_california_housing
import pandas as pd
from sklearn.model_selection import train_test_split

# Load data
data = fetch_california_housing(as_frame=True)
df = data.frame.rename(columns={"MedHouseVal": "target"})
train_df, test_df = train_test_split(df, test_size=0.2, random_state=42)

train_data = TabularDataset(train_df)
test_data  = TabularDataset(test_df)

# AutoGluon: try all models, ensemble best
predictor = TabularPredictor(
    label="target",
    eval_metric="rmse",
    path="./autogluon_models",
).fit(
    train_data,
    time_limit=300,           # 5-minute budget
    presets="best_quality",   # or "medium_quality" for speed
    # Tries: LightGBM, XGBoost, CatBoost, RF, ExtraTrees, NNs, weighted ensembles
)

# Results
leaderboard = predictor.leaderboard(test_data, silent=True)
print(leaderboard[["model", "score_test", "score_val", "fit_time"]].head(10))

# Best model predictions
y_pred = predictor.predict(test_data.drop(columns=["target"]))

# Feature importance (from the best model)
importance = predictor.feature_importance(test_data)
print("
Feature importance:
", importance.head(10))

# ─── AutoML with H2O ─────────────────────────────────────────
import h2o
from h2o.automl import H2OAutoML

h2o.init()
h2o_train = h2o.H2OFrame(train_df)
h2o_test  = h2o.H2OFrame(test_df.drop(columns=["target"]))

aml = H2OAutoML(max_runtime_secs=300, seed=42)
aml.train(x=list(df.drop(columns=["target"]).columns),
          y="target", training_frame=h2o_train)

print(aml.leaderboard.head())
best_model = aml.leader

EDA Tools and Libraries

Tool	Purpose
pandas-profiling / ydata-profiling	Automated EDA report (distributions, correlations, missing values)
sweetviz	Side-by-side train/test comparison report
dtale	Interactive pandas dataframe explorer
missingno	Missing value visualization and patterns
seaborn / matplotlib	Statistical plotting
plotly / altair	Interactive charts
SHAP	Feature importance and interaction analysis post-modeling

# One-line automated EDA report
from ydata_profiling import ProfileReport
profile = ProfileReport(df, title="EDA Report", explorative=True)
profile.to_file("eda_report.html")

Knowledge check

When should you use target encoding instead of one-hot encoding for a categorical feature?

Summary

EDA essentials:

Always check: shape, dtypes, missing values, target distribution, correlations, outliers
Skewed features → log transform; high-cardinality cats → target/frequency encoding
Add missing indicators when imputing

Feature engineering priorities:

Domain-specific features (biggest gain)
Datetime decomposition
Interactions between top features
Polynomial features for linear models

Dimensionality reduction:

PCA: unsupervised, choose $k$ for 95% explained variance; always scale first
LDA: supervised, maximizes class separability — prefer for classification
UMAP: non-linear, fast, good for visualization and as feature extractor
Truncated SVD: for sparse matrices (TF-IDF, document-term)

AutoML tools:

AutoGluon: fastest to state-of-the-art results on tabular data
H2O AutoML: great for production with interpretability needs
Always use AutoML as a baseline, not as the final answer

Next: Recommendation Systems — collaborative filtering, matrix factorization, content-based filtering, and deep learning approaches for personalized recommendations.

← ML Fundamentals