Transparency & Explainability (XAI) — AI Tutorials

Why Explainability Matters

Explainability (or Interpretability) allows humans to understand, verify, and trust AI decisions. It is not merely a technical nicety — it is a legal, ethical, and operational requirement:

Legal Requirements

GDPR Article 22 (EU): Individuals have the right to not be subject to purely automated decisions with significant effects. When such decisions are made, individuals have the right to meaningful information about the logic involved.
EU AI Act: High-risk AI systems must be "sufficiently transparent" to enable operators and users to interpret outputs.
US ECOA (Equal Credit Opportunity Act): Lenders must provide specific reasons for adverse credit decisions.
HIPAA: Healthcare AI must be explainable to clinicians who rely on it.

Operational Benefits

Beyond compliance, explainability enables:

Debugging: understanding why a model fails on specific inputs
Trust: clinicians, judges, and loan officers need to understand recommendations before acting on them
Bias detection: SHAP values reveal if protected attributes are influencing predictions
Regulatory audit: provide evidence trails for model behavior under scrutiny
Model improvement: feature importance guides feature engineering

The Explanation Landscape

Dimension	Options
Scope	Local (one prediction) vs Global (overall model)
Applicability	Model-agnostic vs Model-specific
Form	Feature importance, decision rules, saliency maps, counterfactuals, examples
Audience	Technical (developers) vs Non-technical (end users, regulators)

SHAP: Shapley Additive Explanations

For a model $f$ and input $x$ , SHAP decomposes the prediction as:

$f(x) = \phi_0 + \sum_{j=1}^{M} \phi_j$

$\phi_0 = E[f(x)]$ : the baseline (mean prediction)
$\phi_j$ : the SHAP value for feature $j$ — its contribution to the prediction

The Shapley value for feature $j$ is computed by averaging the marginal contribution over all possible feature orderings $\mathcal{S}$ :

$\phi_j = \sum_{\mathcal{S} \subseteq \mathcal{F} \setminus \{j\}} \frac{|\mathcal{S}|!(|\mathcal{F}|-|\mathcal{S}|-1)!}{|\mathcal{F}|!} \left[f_{\mathcal{S} \cup \{j\}}(x) - f_{\mathcal{S}}(x)\right]$

Interpretation: positive $\phi_j$ means feature $j$ pushed the prediction above the baseline; negative means it pushed it below.

SHAP Variants

Variant	Best For	Speed
TreeSHAP	Tree-based models (XGBoost, LightGBM, Random Forest)	Very fast (exact, polynomial)
DeepSHAP	Neural networks (deep learning)	Fast (approximate)
KernelSHAP	Any model (model-agnostic)	Slow (exponential → sampled)
LinearSHAP	Linear models	Exact, instant
GradientSHAP	Neural networks	Fast

SHAP Visualizations

Force plot: shows how each feature pushed a single prediction above/below baseline
Summary plot (beeswarm): shows all features' SHAP distributions across the dataset — reveals global importance and nonlinear effects
Dependence plot: shows SHAP value for feature $j$ vs its actual value, revealing interactions
Waterfall plot: step-by-step explanation for one prediction

LIME: Local Interpretable Model-agnostic Explanations

LIME (Ribeiro et al., 2016) explains a single prediction by fitting a local linear model around the instance:

Perturb the instance: generate $N$ samples near $x$ by randomly masking/substituting features
Query the black-box model on each perturbed sample to get predictions
Weight samples by proximity to $x$ (exponential kernel: $w_i = \exp(-d_i^2 / \sigma^2)$ )
Fit a sparse linear model on the weighted perturbed samples
Report the linear model's coefficients as the explanation

Advantages: model-agnostic (works for any classifier), supports text, image, and tabular data Disadvantages: explanations are unstable (different runs may yield different explanations for same input), does not capture global model behavior

Counterfactual Explanations

A counterfactual explanation answers: "What is the minimum change to the input that would change the prediction?"

"Your loan was denied. If your annual income were €5,000 higher, it would have been approved."

This is actionable — it tells the user what they can change. Under GDPR, counterfactual explanations are considered a strong form of the "right to explanation."

DiCE (Diverse Counterfactual Explanations): generates diverse counterfactuals that span the feature space, giving multiple actionable recourse paths.

SHAP, LIME, and Counterfactual Explanationspython

# pip install shap lime dice-ml xgboost scikit-learn

# ─── 1. SHAP with XGBoost ────────────────────────────────────
import shap
import xgboost as xgb
import numpy as np
import pandas as pd
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

# Load dataset
data = load_breast_cancer()
X = pd.DataFrame(data.data, columns=data.feature_names)
y = data.target

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Train XGBoost
model = xgb.XGBClassifier(n_estimators=100, max_depth=4, random_state=42, eval_metric='logloss')
model.fit(X_train, y_train)
print(f"Model accuracy: {model.score(X_test, y_test):.4f}")

# Create SHAP explainer (TreeSHAP — fast and exact for tree models)
explainer = shap.TreeExplainer(model)
shap_values = explainer.shap_values(X_test)
# For binary classification, shap_values shape: (n_samples, n_features)

print("\nGlobal Feature Importance (mean |SHAP|):")
mean_shap = np.abs(shap_values).mean(axis=0)
importance_df = pd.DataFrame({
    'feature': X.columns,
    'mean_shap': mean_shap
}).sort_values('mean_shap', ascending=False)
print(importance_df.head(10).to_string(index=False))

# Explain a single prediction
sample_idx = 0
sample = X_test.iloc[sample_idx:sample_idx+1]
pred = model.predict_proba(sample)[0, 1]
shap_single = shap_values[sample_idx]

print(f"\nLocal Explanation for sample {sample_idx}:")
print(f"  Prediction: {pred:.4f} (base value: {explainer.expected_value:.4f})")
local_df = pd.DataFrame({
    'feature': X.columns,
    'value': sample.values[0],
    'shap_value': shap_single,
}).sort_values('shap_value', key=abs, ascending=False)
print(local_df.head(8).to_string(index=False))

# Visualization (saves plots if matplotlib available)
# shap.summary_plot(shap_values, X_test)  # beeswarm
# shap.waterfall_plot(shap.Explanation(shap_single, explainer.expected_value, X_test.iloc[0], feature_names=list(X.columns)))

# ─── 2. SHAP for Neural Networks ─────────────────────────────
import torch
import torch.nn as nn

class SimpleNet(nn.Module):
    def __init__(self, in_dim):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(in_dim, 64), nn.ReLU(),
            nn.Linear(64, 32), nn.ReLU(),
            nn.Linear(32, 1), nn.Sigmoid()
        )
    def forward(self, x):
        return self.net(x)

scaler = StandardScaler()
X_tr_scaled = torch.FloatTensor(scaler.fit_transform(X_train))
X_te_scaled = torch.FloatTensor(scaler.transform(X_test))

nn_model = SimpleNet(X_train.shape[1])
optimizer = torch.optim.Adam(nn_model.parameters(), lr=1e-3)
criterion = nn.BCELoss()

for epoch in range(50):
    nn_model.train()
    out = nn_model(X_tr_scaled).squeeze()
    loss = criterion(out, torch.FloatTensor(y_train.values))
    optimizer.zero_grad(); loss.backward(); optimizer.step()

# GradientSHAP for neural network
background = X_te_scaled[:50]  # reference dataset
grad_explainer = shap.GradientExplainer(nn_model, background)
nn_shap_values = grad_explainer.shap_values(X_te_scaled[:20])
print(f"\nNeural Net SHAP values shape: {np.array(nn_shap_values).shape}")

# ─── 3. LIME for Tabular Data ────────────────────────────────
from lime import lime_tabular

lime_explainer = lime_tabular.LimeTabularExplainer(
    training_data=X_train.values,
    feature_names=list(X.columns),
    class_names=['malignant', 'benign'],
    mode='classification',
    discretize_continuous=True,
    random_state=42,
)

# Explain one prediction
sample_lime = X_test.values[0]
exp = lime_explainer.explain_instance(
    data_row=sample_lime,
    predict_fn=model.predict_proba,
    num_features=8,
    num_samples=1000,
)

print("\nLIME Explanation (top 8 features):")
for feature, weight in exp.as_list():
    direction = "▲" if weight > 0 else "▼"
    print(f"  {direction} {weight:+.4f}  {feature}")

# ─── 4. Counterfactual Explanations with DiCE ─────────────────
import dice_ml

# Wrap data and model for DiCE
d = dice_ml.Data(
    dataframe=pd.concat([X_train, pd.Series(y_train, name='target')], axis=1),
    continuous_features=list(X.columns),
    outcome_name='target',
)
m = dice_ml.Model(model=model, backend='sklearn')
exp_dice = dice_ml.Dice(d, m, method='random')

# Generate counterfactuals for a denied (malignant) prediction
query = X_test.iloc[0:1]
if model.predict(query)[0] == 0:  # malignant = denied
    cf = exp_dice.generate_counterfactuals(
        query_instances=query,
        total_CFs=3,
        desired_class='opposite',
    )
    print("\nCounterfactual Explanations (what changes would flip the prediction):")
    cf.visualize_as_dataframe(show_only_changes=True)

# ─── 5. Model Card Generation ────────────────────────────────
model_card = {
    "model_details": {
        "name": "Breast Cancer Classifier v1.0",
        "type": "XGBoost Binary Classifier",
        "version": "1.0.0",
        "date": "2025-01-15",
        "authors": ["ML Team"],
        "license": "Internal Use Only",
    },
    "intended_use": {
        "primary_use_cases": ["Assist radiologists in classifying breast mass features"],
        "out_of_scope": ["Standalone diagnostic without physician review", "Non-mammography data"],
    },
    "training_data": {
        "dataset": "UCI Breast Cancer Wisconsin (Diagnostic)",
        "size": "569 samples",
        "features": "30 computed from digitized images of fine needle aspirates",
        "known_limitations": ["Single institution data", "Historical data may not reflect current imaging"],
    },
    "evaluation_results": {
        "accuracy": f"{model.score(X_test, y_test):.4f}",
        "top_features_by_shap": importance_df.head(5)['feature'].tolist(),
        "fairness_assessment": "Protected attributes not present; recommend demographic sub-group analysis with real patient data",
    },
    "ethical_considerations": [
        "Must be used with physician oversight (ISO 42001 A.9.4)",
        "Not a substitute for clinical judgment",
        "Explanations via SHAP must be reviewed before communicating to patients",
    ],
    "caveats": [
        "Performance may degrade on data from different imaging equipment",
        "Model validated on US population; performance on other demographics unknown",
    ],
}
import json
print("\nModel Card (ISO 42001 A.8.2 Transparency):")
print(json.dumps(model_card, indent=2))

Model Cards and Datasheets

Model Cards (Mitchell et al., 2019)

A model card is a short document accompanying a trained model that documents its intended use, performance characteristics, and ethical considerations. Required by ISO 42001 A.8.2 (Information for interested parties).

Mandatory sections:

Model details: name, version, type, training date, authors
Intended use: primary use cases, out-of-scope uses
Factors: relevant groups, instrumentation, environmental factors
Metrics: what metrics were evaluated and why
Evaluation data: what data was used for evaluation, and any gaps
Training data: data source, size, known limitations
Quantitative analyses: disaggregated performance by subgroup
Ethical considerations: potential harms, sensitive uses
Caveats and recommendations: known failure modes, deployment constraints

Datasheets for Datasets (Gebru et al., 2018)

A datasheet documents training datasets — complementing model cards:

Motivation: why was the dataset created?
Composition: what is in it? Is there any sensitive data?
Collection process: how was data collected? Who consented?
Preprocessing: what cleaning was applied?
Uses: recommended and discouraged uses
Distribution: how is it distributed, under what license?
Maintenance: who maintains it and how?

ISO 42001 A.8.2 Transparency Requirements

The standard requires organizations to provide to interested parties (users, affected individuals, regulators):

What the AI system does and its limitations
How automated decisions are made (at an appropriate level of detail)
How to seek human review or redress
Contact for questions about the AI system

Knowledge check

A data scientist uses SHAP to explain a model's predictions and finds that a feature called "neighborhood_code" has a high SHAP value. What concern should this raise?

Summary

Explainability is legally required under GDPR Art.22, EU AI Act, and ECOA for automated decisions — and operationally essential for debugging, auditing, and trust
SHAP provides theoretically grounded, consistent explanations satisfying all Shapley axioms; TreeSHAP is fast for tree models, KernelSHAP works on any model
LIME builds a local linear surrogate around a single prediction — model-agnostic but less stable than SHAP
Counterfactual explanations (DiCE) provide actionable recourse: "what would need to change for a different outcome?"
Model cards and datasheets are the primary transparency artifacts required by ISO 42001 A.8.2 — document before deployment
Proxy discrimination is revealed through SHAP: high-importance features correlated with protected attributes require immediate fairness investigation

Next: Accountability & Governance — establishing clear responsibility, audit processes, and organizational structures.

← Responsible AI