Skip to content
SDB
Responsible AI

Chapter 06 · advanced · 33 min

Privacy & Security

Differential privacy, federated learning, model attacks, PII protection, and GDPR compliance for AI systems

Subhendu Datta BhowmikAI Tutorials

Privacy Risks in AI

AI systems introduce novel privacy risks that go beyond traditional data protection:

AttackWhat the Adversary Can LearnExample
Membership inferenceWhether a specific record was in the training dataAttacker learns a person was in a medical trial dataset
Model inversionReconstruct training data samples from model outputsReconstruct faces of people in facial recognition training set
Model extractionClone a black-box model using query-response pairsSteal a proprietary model via its API
Data poisoningIntroduce backdoors during trainingAdversary corrupts data to make model fail on specific inputs
Property inferenceInfer aggregate properties of training dataInfer demographic makeup of a training dataset
Gradient inversionReconstruct training data from shared gradientsCritical for federated learning: recover images from gradients

Why Standard Data Anonymization Is Insufficient

Traditional techniques like pseudonymization, k-anonymity, and l-diversity are insufficient for ML:

  • Linkage attacks: de-anonymize records by combining the ML model's outputs with auxiliary information
  • Inference attacks: ML models encode statistical patterns of training data — even "anonymized" training data can be reconstructed
  • Temporal re-identification: language models trained on text can regurgitate memorized sequences including PII

ISO 42001 — A.7.2 (Data Governance for AI)

Control A.7.2 requires:

  • Data minimization: collect only the data necessary for the AI system's purpose
  • Purpose limitation: use data only for the stated AI purpose
  • Data quality: assess accuracy, completeness, representativeness of training data
  • Retention and deletion: AI training data must be governed by retention policies
  • Third-party data: ensure appropriate contractual data processing agreements

Differential Privacy

The Privacy-Utility Trade-off

DP injects calibrated noise into computations to prevent individual records from influencing outputs:

f~(D)=f(D)+Laplace(0,Δf/ϵ)\tilde{f}(D) = f(D) + \text{Laplace}(0, \Delta f / \epsilon)

where Δf\Delta f is the global sensitivity (maximum change in ff when one record changes), and ϵ\epsilon is the privacy budget.

Interpreting ε:

  • ϵ=0\epsilon = 0: perfect privacy (pure noise — useless output)
  • ϵ=1\epsilon = 1: strong practical privacy (academic/medical standard)
  • ϵ=10\epsilon = 10: moderate (industry common for aggregate statistics)
  • ϵ\epsilon \to \infty: no privacy

Composability: kk mechanisms each with budget ϵi\epsilon_i compose to iϵi\sum_i \epsilon_i total.

DP-SGD (Differentially Private Stochastic Gradient Descent)

DP-SGD (Abadi et al., 2016) trains neural networks with differential privacy:

  1. Clip gradients per sample: g~i=gi/max(1,gi2/C)\tilde{g}_i = g_i / \max(1, \|g_i\|_2 / C) — limits sensitivity
  2. Add noise: G~=1B(ig~i+N(0,σ2C2I))\tilde{G} = \frac{1}{B} \left( \sum_i \tilde{g}_i + \mathcal{N}(0, \sigma^2 C^2 \mathbf{I}) \right)
  3. Update: θθηG~\theta \leftarrow \theta - \eta \tilde{G}

The moments accountant tracks privacy loss accurately across training steps.

Practical cost of DP: MNIST classifier accuracy drops from ~99.5% to ~98.5% at ε=10. For more complex tasks, the accuracy-privacy trade-off is more severe.

Federated Learning

Federated Learning (FL) trains models across distributed devices without centralizing data:

Central Server (aggregator)
  │
  ├─ Send global model weights → Device 1 (hospital A)
  ├─ Send global model weights → Device 2 (hospital B)
  └─ Send global model weights → Device 3 (hospital C)
       │
       Each device:
         1. Train on local data for k steps
         2. Compute weight update ΔW
         3. Send ΔW (not raw data) to server
       │
  Server: Aggregate: W_global ← W_global + avg(ΔW_1, ΔW_2, ΔW_3)
  │
  └─ Repeat until convergence

FedAvg (McMahan et al., 2017): simple weighted average of client updates:

Wt+1=k=1KnknWt+1kW_{t+1} = \sum_{k=1}^{K} \frac{n_k}{n} W_{t+1}^k

Privacy note: Sharing gradients is NOT perfectly private — gradient inversion attacks can reconstruct training data. Secure aggregation and DP on gradients are recommended additions.

FL Challenges

ChallengeDescriptionMitigation
Statistical heterogeneityData distributions differ across clientsFedProx, personalization
System heterogeneityDifferent compute/bandwidth per deviceAsynchronous FL, client selection
Communication costGradient sizes can be largeGradient compression, sparsification
Gradient inversionGradients leak training dataDP-SGD on gradients, secure aggregation
Differential Privacy, Membership Inference Defense, and PII Redactionpython
# pip install opacus torch presidio-analyzer presidio-anonymizer

# ─── 1. DP-SGD with Opacus (PyTorch) ─────────────────────────
import torch
import torch.nn as nn
import torch.optim as optim
from torch.utils.data import DataLoader, TensorDataset
from opacus import PrivacyEngine
from opacus.validators import ModuleValidator
import numpy as np

# Simple model
class LinearClassifier(nn.Module):
    def __init__(self, input_dim=20, num_classes=2):
        super().__init__()
        self.fc = nn.Sequential(
            nn.Linear(input_dim, 64), nn.ReLU(),
            nn.Linear(64, num_classes)
        )
    def forward(self, x):
        return self.fc(x)

# Synthetic dataset
X = torch.randn(1000, 20)
y = (X[:, 0] + X[:, 1] > 0).long()
dataset = TensorDataset(X, y)
loader = DataLoader(dataset, batch_size=64, shuffle=True)

model = LinearClassifier()
optimizer = optim.Adam(model.parameters(), lr=1e-3)
criterion = nn.CrossEntropyLoss()

# Make model compatible with Opacus (fixes BatchNorm → GroupNorm)
model = ModuleValidator.fix(model)
errors = ModuleValidator.validate(model, strict=False)
print(f"Model validation errors: {errors}")

# Attach DP engine
privacy_engine = PrivacyEngine()
model, optimizer, loader = privacy_engine.make_private_with_epsilon(
    module=model,
    optimizer=optimizer,
    data_loader=loader,
    epochs=5,
    target_epsilon=3.0,    # Privacy budget
    target_delta=1e-5,     # Failure probability
    max_grad_norm=1.0,     # Gradient clipping threshold
)

print(f"\nDP-SGD Configuration:")
print(f"  Target ε = 3.0, δ = 1e-5")
print(f"  Noise multiplier σ = {optimizer.noise_multiplier:.4f}")
print(f"  Max gradient norm C = 1.0")

# Training loop
for epoch in range(3):
    model.train()
    total_loss = 0
    for batch_x, batch_y in loader:
        optimizer.zero_grad()
        output = model(batch_x)
        loss = criterion(output, batch_y)
        loss.backward()
        optimizer.step()
        total_loss += loss.item()

    epsilon = privacy_engine.get_epsilon(delta=1e-5)
    print(f"  Epoch {epoch+1}: loss={total_loss/len(loader):.4f}, ε spent={epsilon:.4f}")

print(f"\nFinal privacy guarantee: ε={privacy_engine.get_epsilon(1e-5):.3f}, δ=1e-5")

# ─── 2. Membership Inference Attack + Detection ───────────────
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score

# Train a target model
X_data = np.random.randn(2000, 20)
y_data = (X_data[:, 0] > 0).astype(int)
X_train_mi, X_shadow, y_train_mi, y_shadow = train_test_split(X_data, y_data, test_size=0.5)
target_model = RandomForestClassifier(n_estimators=50, random_state=42)
target_model.fit(X_train_mi, y_train_mi)

# Simple membership inference: training samples have higher confidence
def mi_attack_confidence(model, X_members, X_non_members):
    """Naive MI attack: members → high confidence, non-members → lower confidence."""
    member_conf = model.predict_proba(X_members).max(axis=1)
    nonmember_conf = model.predict_proba(X_non_members).max(axis=1)

    all_conf = np.concatenate([member_conf, nonmember_conf])
    all_labels = np.concatenate([np.ones(len(X_members)), np.zeros(len(X_non_members))])
    auc = roc_auc_score(all_labels, all_conf)
    print(f"  MI Attack AUC: {auc:.4f} (0.5=random=perfectly private, 1.0=full membership leaked)")
    print(f"  Member avg conf:     {member_conf.mean():.4f}")
    print(f"  Non-member avg conf: {nonmember_conf.mean():.4f}")
    return auc

print("\nMembership Inference Attack:")
mi_attack_confidence(target_model, X_train_mi[:100], X_shadow[:100])

# ─── 3. Simulated Federated Learning ─────────────────────────
class FederatedServer:
    def __init__(self, global_model):
        self.global_weights = {k: v.clone() for k, v in global_model.state_dict().items()}

    def aggregate(self, client_updates, client_sizes):
        """FedAvg: weighted average of client model updates."""
        total = sum(client_sizes)
        new_weights = {}
        for key in self.global_weights:
            new_weights[key] = sum(
                (n / total) * updates[key]
                for updates, n in zip(client_updates, client_sizes)
            )
        self.global_weights = new_weights
        return self.global_weights

def client_train(global_weights, local_X, local_y, local_epochs=5, lr=0.01):
    """Train a local model on client data and return updated weights."""
    model = LinearClassifier()
    model.load_state_dict(global_weights)
    optimizer = optim.SGD(model.parameters(), lr=lr)
    criterion = nn.CrossEntropyLoss()
    dataset = TensorDataset(
        torch.FloatTensor(local_X),
        torch.LongTensor(local_y)
    )
    loader = DataLoader(dataset, batch_size=32, shuffle=True)
    for _ in range(local_epochs):
        for bx, by in loader:
            optimizer.zero_grad()
            loss = criterion(model(bx), by)
            loss.backward()
            optimizer.step()
    return {k: v.detach().clone() for k, v in model.state_dict().items()}, len(local_X)

# Simulate 3 clients (e.g., 3 hospitals with private patient data)
global_model = LinearClassifier()
server = FederatedServer(global_model)

client_data = [
    (np.random.randn(300, 20), (np.random.randn(300) > 0).astype(int)),
    (np.random.randn(200, 20), (np.random.randn(200) > 0).astype(int)),
    (np.random.randn(400, 20), (np.random.randn(400) > 0).astype(int)),
]

print("\nFederated Learning Simulation (3 clients, 3 rounds):")
for round_num in range(3):
    updates, sizes = [], []
    for i, (cx, cy) in enumerate(client_data):
        client_weights, n = client_train(server.global_weights, cx, cy)
        updates.append(client_weights)
        sizes.append(n)
    server.global_weights = server.aggregate(updates, sizes)
    print(f"  Round {round_num+1}: aggregated {sum(sizes)} samples from {len(client_data)} clients")

# ─── 4. PII Detection and Anonymization ──────────────────────
from presidio_analyzer import AnalyzerEngine
from presidio_anonymizer import AnonymizerEngine
from presidio_anonymizer.entities import OperatorConfig

analyzer = AnalyzerEngine()
anonymizer = AnonymizerEngine()

texts_with_pii = [
    "Patient John Smith (DOB: 15/03/1985) was diagnosed at john.smith@email.com. His SSN is 123-45-6789.",
    "Call Dr. Sarah Johnson at +1-555-0198 regarding claim #AB12345 for Mary Williams.",
    "Credit card 4532-1234-5678-9012 was used at 123 Main St, New York, NY 10001.",
]

print("\nPII Detection and Anonymization:")
for text in texts_with_pii:
    results = analyzer.analyze(
        text=text,
        entities=["PERSON", "EMAIL_ADDRESS", "PHONE_NUMBER", "US_SSN",
                  "CREDIT_CARD", "DATE_TIME", "LOCATION", "US_DRIVER_LICENSE"],
        language="en"
    )
    anonymized = anonymizer.anonymize(
        text=text,
        analyzer_results=results,
        operators={
            "PERSON": OperatorConfig("replace", {"new_value": "[PERSON]"}),
            "EMAIL_ADDRESS": OperatorConfig("replace", {"new_value": "[EMAIL]"}),
            "PHONE_NUMBER": OperatorConfig("replace", {"new_value": "[PHONE]"}),
            "US_SSN": OperatorConfig("replace", {"new_value": "[SSN]"}),
            "CREDIT_CARD": OperatorConfig("replace", {"new_value": "[CC]"}),
            "DATE_TIME": OperatorConfig("replace", {"new_value": "[DATE]"}),
            "LOCATION": OperatorConfig("replace", {"new_value": "[LOCATION]"}),
        }
    )
    print(f"\n  Original:    {text[:80]}")
    print(f"  Anonymized:  {anonymized.text[:80]}")
    detected = [(r.entity_type, text[r.start:r.end]) for r in results]
    print(f"  Detected PII: {detected}")

GDPR Compliance Framework for AI

Key GDPR Articles for AI Systems

ArticleRequirementAI Implication
Art. 5Data minimization, purpose limitationTrain on minimum necessary data; don't repurpose without consent
Art. 13/14Transparency (right to information)Disclose that AI processes their data and how
Art. 17Right to erasure (right to be forgotten)Must be able to retrain/remove data influence
Art. 21Right to objectUsers can object to automated profiling
Art. 22Right not to be subject to automated decisionsProvide human review option for significant decisions
Art. 25Privacy by designBuild privacy controls into systems from the start
Art. 35Data Protection Impact Assessment (DPIA)Required for high-risk processing — equivalent to AIIA

The Right to Be Forgotten Challenge

Machine Unlearning is an open research problem: GDPR Art.17 says users can request deletion of their data — but for ML models, data is encoded in weights across millions of parameters.

Current approaches:

  1. Retraining from scratch without the deleted records (expensive but exact)
  2. Data removal approximation: techniques like SISA (Sharded, Isolated, Sliced, and Aggregated) training to enable efficient partial retraining
  3. Gradient ascent unlearning: fine-tune the model to "unlearn" specific samples (approximate but fast)

Privacy-by-Design Checklist (ISO 42001 A.7.2 + GDPR Art.25)

  • Data minimization: only collect features needed for the stated task
  • Purpose limitation: document how data will be used before collection
  • Consent: collect and document explicit consent for training data use
  • PII scanning: run automated PII detection on all training data
  • Pseudonymization: replace direct identifiers before training
  • Retention schedule: define and automate data deletion timelines
  • Differential privacy: apply DP-SGD for sensitive domains (medical, financial)
  • Model audit: run membership inference probes before deployment
  • DPIA/AIIA: complete for any high-risk processing

Knowledge check

A company trains a language model on customer support chats and deploys it. A customer requests deletion of their data under GDPR Art.17. What is the most technically challenging aspect of complying?

Summary

  • AI systems face unique privacy attacks: membership inference, model inversion, model extraction, and gradient inversion — none of which are addressed by traditional anonymization
  • Differential Privacy provides formal guarantees: ε-DP means any output changes by at most eϵe^\epsilon when one person's data is added/removed; DP-SGD applies this to neural network training
  • Federated Learning trains models without centralizing data but is not sufficient alone — gradient inversion attacks require secure aggregation + DP on gradients
  • PII detection and anonymization (Presidio, Faker) must be applied to training data before ML model development
  • GDPR Art.17 (right to erasure) creates a machine unlearning challenge — models must either be retrained or use approximation techniques to remove data influence
  • ISO 42001 A.7.2 mandates data minimization, purpose limitation, and appropriate data governance throughout the AI lifecycle

Next: Human-Centricity & Empathy — keeping humans in the loop and designing AI that respects human agency.

Responsible AI