Model Serving & Inference — AI Tutorials

Model Serving Fundamentals

Training a model is one thing — serving it reliably at scale is another. Model serving is the process of exposing a trained model as a service that accepts inputs and returns predictions.

Serving Patterns

Pattern	Description	When to Use
Real-time (online)	Synchronous, low-latency per-request inference	Fraud detection, search ranking
Batch	Asynchronous processing of large datasets	Overnight scoring, report generation
Streaming	Continuous inference on event streams	Clickstream analysis, IoT
Edge	On-device inference without network	Mobile apps, embedded systems

Key Metrics

Latency (p50, p95, p99): time from request to response
Throughput: requests per second the server can handle
TTFT: time to first token (for LLMs)
Model load time: startup cost when scaling horizontally

FastAPI for Custom Model Serving

FastAPI is the go-to framework for wrapping any ML model in a Python REST API. It's fast, type-safe, and auto-generates OpenAPI docs.

Production FastAPI Serving with Lifecycle Managementpython

from contextlib import asynccontextmanager
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
import joblib
import numpy as np
from prometheus_client import Counter, Histogram, make_asgi_app
import time

# --- Pydantic schemas ---
class PredictionRequest(BaseModel):
    features: list[float]

class PredictionResponse(BaseModel):
    prediction: int
    probability: float
    model_version: str

# --- Metrics ---
PREDICTION_COUNT = Counter("predictions_total", "Total predictions", ["result"])
PREDICTION_LATENCY = Histogram("prediction_latency_seconds", "Prediction latency")

# --- Model lifecycle ---
model_store: dict = {}

@asynccontextmanager
async def lifespan(app: FastAPI):
    # Load on startup
    model_store["model"] = joblib.load("models/fraud_detector.pkl")
    model_store["version"] = "2.1.0"
    yield
    # Cleanup on shutdown
    model_store.clear()

app = FastAPI(title="Fraud Detector API", lifespan=lifespan)

# Mount Prometheus metrics endpoint
app.mount("/metrics", make_asgi_app())

@app.post("/predict", response_model=PredictionResponse)
async def predict(request: PredictionRequest):
    start = time.time()
    try:
        X = np.array(request.features).reshape(1, -1)
        model = model_store["model"]
        prob = float(model.predict_proba(X)[0, 1])
        pred = int(prob >= 0.5)

        PREDICTION_COUNT.labels(result=str(pred)).inc()
        PREDICTION_LATENCY.observe(time.time() - start)

        return PredictionResponse(
            prediction=pred,
            probability=prob,
            model_version=model_store["version"],
        )
    except Exception as e:
        raise HTTPException(status_code=500, detail=str(e))

@app.get("/health")
async def health():
    return {"status": "ok", "model_loaded": "model" in model_store}

ONNX: Cross-Framework Optimized Inference

ONNX (Open Neural Network Exchange) is an open format for ML models. Converting to ONNX lets you:

Run PyTorch models with ONNX Runtime (2–4× faster than PyTorch eager mode)
Deploy to any ONNX-compatible runtime (TensorRT, OpenVINO, CoreML)
Serve models without the training framework dependency

Export PyTorch to ONNX and Run with ONNX Runtimepython

import torch
import onnx
import onnxruntime as ort
import numpy as np

# --- Export ---
model = MyPyTorchModel()
model.eval()

dummy_input = torch.randn(1, 128)  # batch_size=1, features=128

torch.onnx.export(
    model,
    dummy_input,
    "models/fraud_detector.onnx",
    opset_version=17,
    input_names=["features"],
    output_names=["logits"],
    dynamic_axes={"features": {0: "batch_size"}, "logits": {0: "batch_size"}},
)

# Validate the exported model
onnx_model = onnx.load("models/fraud_detector.onnx")
onnx.checker.check_model(onnx_model)

# --- Inference with ONNX Runtime ---
sess_options = ort.SessionOptions()
sess_options.intra_op_num_threads = 4
sess_options.execution_mode = ort.ExecutionMode.ORT_PARALLEL

session = ort.InferenceSession(
    "models/fraud_detector.onnx",
    sess_options,
    providers=["CUDAExecutionProvider", "CPUExecutionProvider"],
)

inputs = np.random.randn(32, 128).astype(np.float32)  # batch of 32
outputs = session.run(["logits"], {"features": inputs})
logits = outputs[0]  # shape: (32, 1)

TorchServe

TorchServe is the official PyTorch model server. It handles model versioning, A/B testing, batching, and metrics out of the box.

Architecture:

Frontend: HTTP/gRPC endpoint (management + inference APIs)
Backend: worker processes that load and run models
Model store: directory of .mar (Model Archive) files

TorchServe: Package and Deploybash

# 1. Create a custom handler
# handler.py defines preprocess, inference, postprocess methods

# 2. Package the model into a .mar archive
torch-model-archiver \
  --model-name fraud_detector \
  --version 2.1 \
  --model-file model.py \
  --serialized-file models/fraud_detector.pt \
  --handler handler.py \
  --export-path model_store

# 3. Start TorchServe
torchserve \
  --start \
  --ncs \
  --model-store model_store \
  --models fraud_detector=fraud_detector.mar \
  --ts-config config.properties

# 4. Inference
curl -X POST http://localhost:8080/predictions/fraud_detector \
  -H "Content-Type: application/json" \
  -d '{"features": [0.1, 0.4, 1.2, ...]}'

NVIDIA Triton Inference Server

Triton is the highest-performance option for GPU-accelerated serving. It supports TensorRT, ONNX, TensorFlow, PyTorch, and Python backends in a single server.

Key features:

Dynamic batching: automatically batches requests to maximize GPU utilization
Concurrent model execution: multiple models share GPU memory
Model ensembles: chain models together in a pipeline
BLS (Business Logic Scripting): custom Python pre/post-processing in the pipeline

Triton Model Repository Layouttext

model_repository/
└── fraud_detector/
    ├── config.pbtxt          # model configuration
    └── 1/
        └── model.onnx        # model file (version 1)

# config.pbtxt
name: "fraud_detector"
backend: "onnxruntime"
max_batch_size: 64
input [
  {
    name: "features"
    data_type: TYPE_FP32
    dims: [128]
  }
]
output [
  {
    name: "logits"
    data_type: TYPE_FP32
    dims: [1]
  }
]
dynamic_batching {
  preferred_batch_size: [16, 32, 64]
  max_queue_delay_microseconds: 5000
}

Latency Optimization Techniques

1. Quantization

Reduce model precision from FP32 → INT8 or FP16. Cuts memory by 2–4× and speeds up inference.

2. Dynamic Batching

Group multiple requests into a single batch. GPU utilization jumps from 10% (single requests) to 80%+ (batched).

3. Model Compilation

TorchScript: compile PyTorch to a serializable, optimizable graph
torch.compile: JIT compilation with Triton kernels (PyTorch 2.0+)
TensorRT: NVIDIA's optimizer for maximum GPU throughput

4. Caching

Cache predictions for identical or near-identical inputs (semantic caching for LLMs).

Model Quantization with ONNX Runtimepython

from onnxruntime.quantization import quantize_dynamic, QuantType

# Dynamic INT8 quantization (no calibration data needed)
quantize_dynamic(
    model_input="models/fraud_detector.onnx",
    model_output="models/fraud_detector_int8.onnx",
    weight_type=QuantType.QInt8,
)

# Compare sizes
import os
fp32_size = os.path.getsize("models/fraud_detector.onnx") / 1e6
int8_size = os.path.getsize("models/fraud_detector_int8.onnx") / 1e6
print(f"FP32: {fp32_size:.1f} MB → INT8: {int8_size:.1f} MB ({fp32_size/int8_size:.1f}× smaller)")

Knowledge check

What problem does dynamic batching solve in model serving?

Summary

FastAPI is the fastest path to a production REST API for any scikit-learn, ONNX, or custom model
ONNX provides a cross-framework exchange format; ONNX Runtime delivers 2–4× faster CPU inference
TorchServe is the official PyTorch server with model versioning and management APIs
Triton is the highest-performance GPU server, supporting dynamic batching and model ensembles
Quantization (INT8/FP16) cuts model size and latency with minimal accuracy loss
Choose based on your requirements: FastAPI for simplicity, TorchServe for PyTorch, Triton for high-throughput GPU workloads

Next: CI/CD for machine learning — testing models in pipelines and automating retraining.

← MLOps & AI Engineering