Production Monitoring — AI Tutorials

Why Models Degrade in Production

A model is a function learned from historical data. When the world changes, that function becomes stale. This happens through two primary mechanisms:

Data Drift (Covariate Shift)

The distribution of input features P(X) changes, but the relationship between inputs and outputs P(Y|X) stays the same.

Example: a fraud model trained in January sees a surge of transactions from a new country in March. The feature distributions have shifted, but "what constitutes fraud" hasn't changed.

Concept Drift

The relationship P(Y|X) itself changes — the meaning of the target has evolved.

Example: fraudsters adapt their patterns. Transactions that looked legitimate in January now have fraud characteristics in March. The same features now predict different outcomes.

| Type | P(X) changes? | P(Y|X) changes? | Impact | |---|---|---|---| | Data drift | ✅ | ❌ | Model may still work; monitor | | Concept drift | Any | ✅ | Model predictions are wrong; retrain | | Label shift | ❌ | ❌ | Target distribution shifts; recalibrate |

Drift Detection Statistics

Population Stability Index (PSI)

PSI measures how much a distribution has shifted relative to a reference (training) distribution.

$PSI = \sum_{i=1}^{n} (P_{actual,i} - P_{expected,i}) \cdot \ln\left(\frac{P_{actual,i}}{P_{expected,i}}\right)$

PSI Value	Interpretation
< 0.1	No significant change
0.1 – 0.2	Moderate change; investigate
> 0.2	Significant shift; action required

Kolmogorov-Smirnov (KS) Test

Tests whether two samples come from the same distribution. Returns a p-value: if p < 0.05, the distributions are significantly different.

Computing PSI and KS-Testpython

import numpy as np
from scipy import stats

def compute_psi(reference: np.ndarray, current: np.ndarray, buckets: int = 10) -> float:
    """Compute Population Stability Index."""
    # Create bucket boundaries on reference data
    breakpoints = np.percentile(reference, np.linspace(0, 100, buckets + 1))
    breakpoints[0] = -np.inf
    breakpoints[-1] = np.inf

    ref_counts = np.histogram(reference, breakpoints)[0]
    cur_counts = np.histogram(current, breakpoints)[0]

    # Convert to proportions, avoid division by zero
    ref_pct = ref_counts / len(reference)
    cur_pct = cur_counts / len(current)
    ref_pct = np.where(ref_pct == 0, 0.001, ref_pct)
    cur_pct = np.where(cur_pct == 0, 0.001, cur_pct)

    psi = np.sum((cur_pct - ref_pct) * np.log(cur_pct / ref_pct))
    return float(psi)

def check_drift(reference_df, current_df, features: list[str], threshold: float = 0.2):
    alerts = []
    for feature in features:
        psi = compute_psi(reference_df[feature].dropna(), current_df[feature].dropna())
        ks_stat, ks_pval = stats.ks_2samp(reference_df[feature], current_df[feature])

        if psi > threshold or ks_pval < 0.01:
            alerts.append({
                "feature": feature,
                "psi": round(psi, 4),
                "ks_pvalue": round(ks_pval, 4),
                "severity": "high" if psi > 0.25 else "medium",
            })
    return alerts

Evidently AI

Evidently is an open-source library for ML monitoring. It generates interactive HTML reports and JSON metrics for:

Data quality
Data drift (tabular, text, embedding)
Model performance (when ground truth labels are available)
Target drift

Evidently Monitoring Reportpython

import pandas as pd
from evidently.report import Report
from evidently.metric_preset import DataDriftPreset, ClassificationPreset
from evidently.metrics import (
    DatasetDriftMetric,
    ColumnDriftMetric,
    ClassificationQualityMetric,
)

# Reference data (training distribution)
reference = pd.read_parquet("data/features/train.parquet")

# Current production data (last 7 days)
current = pd.read_parquet("data/production/last_7d.parquet")

# --- Data Drift Report ---
drift_report = Report(metrics=[
    DatasetDriftMetric(),
    ColumnDriftMetric(column_name="amount"),
    ColumnDriftMetric(column_name="transaction_count_7d"),
    DataDriftPreset(),
])
drift_report.run(reference_data=reference, current_data=current)
drift_report.save_html("reports/drift_report.html")

# Get drift metrics programmatically
drift_result = drift_report.as_dict()
drift_detected = drift_result["metrics"][0]["result"]["dataset_drift"]
n_drifted = drift_result["metrics"][0]["result"]["number_of_drifted_columns"]
print(f"Drift detected: {drift_detected}, columns drifted: {n_drifted}")

# --- Performance Report (when labels available) ---
perf_report = Report(metrics=[ClassificationPreset()])
perf_report.run(reference_data=reference, current_data=current)
perf_report.save_html("reports/performance_report.html")

Evidently Monitoring Suite (Continuous)python

from evidently.test_suite import TestSuite
from evidently.test_preset import DataDriftTestPreset, DataQualityTestPreset
from evidently.tests import TestColumnDrift, TestShareOfDriftedColumns

# Define the test suite — pass/fail for CI/CD integration
test_suite = TestSuite(tests=[
    DataQualityTestPreset(),
    TestShareOfDriftedColumns(lt=0.3),   # < 30% of columns drifted
    TestColumnDrift(column_name="amount", stattest="psi", stattest_threshold=0.2),
])

test_suite.run(reference_data=reference, current_data=current)
test_suite.save_html("reports/test_suite.html")

results = test_suite.as_dict()
all_passed = results["summary"]["all_passed"]

if not all_passed:
    # Alert and trigger retraining
    send_alert("Drift test suite failed — retraining triggered")
    trigger_retraining("data_drift")

Arize AI and Fiddler AI

For enterprise teams, managed ML observability platforms provide more powerful features:

Arize AI

Embedding drift: monitors NLP embeddings and image embeddings for semantic shift
Explainability: SHAP-based feature importance in production
Performance tracking: latency, prediction distributions, and model score distributions
Alerts: configurable thresholds with PagerDuty/Slack integration

Fiddler AI

Explainable AI monitoring: integrates XAI natively with drift monitoring
Segment monitoring: monitor drift and performance within cohorts (age group, region)
Bias tracking: demographic parity and equalized odds in production
Model comparison: A/B test models and compare performance across segments

Logging Predictions to Arizepython

import arize
from arize.api import Client
from arize.utils.types import ModelTypes, Environments, Schema
import pandas as pd

arize_client = Client(space_key="YOUR_SPACE_KEY", api_key="YOUR_API_KEY")

# Log a batch of production predictions
schema = Schema(
    prediction_id_column_name="transaction_id",
    prediction_label_column_name="predicted_label",
    prediction_score_column_name="fraud_probability",
    actual_label_column_name="actual_label",  # available after ground truth
    feature_column_names=[
        "amount", "transaction_count_7d", "avg_amount_7d", "hour"
    ],
)

response = arize_client.log(
    dataframe=production_df,
    model_id="fraud-detector",
    model_version="2.1.0",
    model_type=ModelTypes.BINARY_CLASSIFICATION,
    environment=Environments.PRODUCTION,
    schema=schema,
)

print(f"Logged {len(production_df)} predictions to Arize")

Monitoring Architecture

A complete production monitoring stack consists of:

Prediction logging: every prediction logged with features, prediction, and (eventually) ground truth
Scheduled drift checks: run daily/hourly, compare to reference window
Alerting: PagerDuty/Slack when PSI > threshold or performance drops
Dashboard: real-time view of drift scores, prediction distributions, and model KPIs
Retraining trigger: automated pipeline dispatch when thresholds are breached (Chapter 6)

Knowledge check

What is the difference between data drift and concept drift?

Summary

Data drift is a shift in P(X); concept drift is a change in P(Y|X) — both degrade model performance but require different responses
PSI (> 0.2 = significant shift) and the KS test (p < 0.05 = significant) are the standard drift statistics
Evidently generates HTML/JSON drift and quality reports and integrates with CI/CD test suites
Arize and Fiddler provide managed ML observability with embedding drift, XAI, and segment monitoring
A complete monitoring stack covers: prediction logging → scheduled drift checks → alerts → retraining triggers

Next: A/B testing and experimentation with statistical power, multi-armed bandits, and holdback groups.

← MLOps & AI Engineering