Data Versioning & Pipelines — AI Tutorials

Why Data Versioning Matters

In software engineering, version control for code is non-negotiable. In ML engineering, data and models must also be versioned. Without it:

You cannot reproduce a past experiment (what data was the model trained on?)
You cannot audit model behavior (what changed between v1 and v2?)
You cannot roll back a bad model update

The challenge: datasets are large binary files that don't belong in Git.

DVC: Data Version Control

DVC (Data Version Control) solves this by storing large files in a remote storage (S3, GCS, Azure Blob, SFTP) while tracking pointers (.dvc files) in Git.

Core Concepts

Git	DVC
Tracks code	Tracks data & models
`.git/` local store	Remote storage (S3, GCS…)
`git commit`	`dvc commit`
`git push`	`dvc push`
`git checkout`	`dvc checkout`

DVC Setup and Basic Workflowbash

# Install
pip install dvc dvc-s3

# Initialize DVC in a Git repo
git init
dvc init

# Configure remote storage
dvc remote add -d myremote s3://my-bucket/dvc-store

# Track a dataset
dvc add data/raw/transactions.parquet
git add data/raw/transactions.parquet.dvc .gitignore
git commit -m "Track raw transactions dataset"
dvc push  # Upload to S3

# Later: reproduce on another machine
git clone <repo>
dvc pull  # Download data from S3

DVC Pipelines

DVC pipelines chain stages together, track dependencies, and only re-run stages when inputs change — like Make, but for ML.

Each stage declares:

deps: input files or directories
outs: output files (automatically tracked)
params: hyperparameters from a YAML config
metrics: evaluation result files

dvc.yaml — Full ML Pipelineyaml

stages:
  prepare:
    cmd: python src/prepare.py
    deps:
      - src/prepare.py
      - data/raw/transactions.parquet
    outs:
      - data/prepared/train.parquet
      - data/prepared/test.parquet

  featurize:
    cmd: python src/featurize.py
    deps:
      - src/featurize.py
      - data/prepared/train.parquet
    params:
      - params.yaml:
          - featurize.window_days
    outs:
      - data/features/train_features.parquet

  train:
    cmd: python src/train.py
    deps:
      - src/train.py
      - data/features/train_features.parquet
    params:
      - params.yaml:
          - train.learning_rate
          - train.n_estimators
    outs:
      - models/model.pkl
    metrics:
      - metrics/train_metrics.json

  evaluate:
    cmd: python src/evaluate.py
    deps:
      - src/evaluate.py
      - models/model.pkl
      - data/prepared/test.parquet
    metrics:
      - metrics/eval_metrics.json:
          cache: false

Running and Comparing Experimentsbash

# Run the full pipeline (only re-runs changed stages)
dvc repro

# Compare experiments
dvc params diff
dvc metrics diff

# Run an experiment with modified params (without changing params.yaml)
dvc exp run --set-param train.learning_rate=0.05

# Show experiment table
dvc exp show

Workflow Orchestration

For production data pipelines that run on schedules or trigger on events, you need an orchestrator. The three most popular options in the ML space are Airflow, Prefect, and Kedro.

Apache Airflow

The industry standard for scheduled workflow orchestration. Pipelines are defined as DAGs (Directed Acyclic Graphs) in Python.

Best for: large teams, complex dependencies, rich ecosystem of operators (Spark, BigQuery, dbt, etc.) Watch out for: steep learning curve, heavy infrastructure, DAGs run on a schedule rather than reacting to events

Airflow DAG — ML Training Pipelinepython

from datetime import datetime
from airflow.decorators import dag, task

@dag(schedule_interval='@daily', start_date=datetime(2024, 1, 1), catchup=False)
def ml_training_pipeline():

    @task
    def extract() -> str:
        # Pull latest data from warehouse
        import pandas as pd
        df = pd.read_sql("SELECT * FROM transactions WHERE date = CURRENT_DATE", conn)
        path = "/tmp/transactions.parquet"
        df.to_parquet(path)
        return path

    @task
    def transform(raw_path: str) -> str:
        import pandas as pd
        df = pd.read_parquet(raw_path)
        df = df.dropna().assign(hour=pd.to_datetime(df['ts']).dt.hour)
        out = "/tmp/features.parquet"
        df.to_parquet(out)
        return out

    @task
    def train(features_path: str) -> None:
        import pandas as pd
        from sklearn.ensemble import GradientBoostingClassifier
        import joblib
        df = pd.read_parquet(features_path)
        X, y = df.drop("label", axis=1), df["label"]
        model = GradientBoostingClassifier().fit(X, y)
        joblib.dump(model, "/models/fraud_detector.pkl")

    raw = extract()
    features = transform(raw)
    train(features)

dag = ml_training_pipeline()

Prefect

A modern alternative to Airflow with a simpler Python-native API and better support for dynamic workflows.

Best for: data science teams that want Python-first workflows without YAML/XML configuration Key features: native async support, automatic retry logic, built-in caching, cloud or self-hosted UI

Prefect Flow — Same Pipelinepython

from prefect import flow, task
from prefect.tasks import task_input_hash
from datetime import timedelta

@task(cache_key_fn=task_input_hash, cache_expiration=timedelta(hours=1))
def extract() -> str:
    # ... fetch data
    return "/tmp/transactions.parquet"

@task(retries=3, retry_delay_seconds=60)
def transform(raw_path: str) -> str:
    # ... feature engineering
    return "/tmp/features.parquet"

@task
def train(features_path: str) -> None:
    # ... train and save model
    pass

@flow(name="daily-fraud-model-training")
def training_pipeline():
    raw = extract()
    features = transform(raw)
    train(features)

if __name__ == "__main__":
    training_pipeline()

Kedro

A framework for creating maintainable, modular data science code. Kedro is not an orchestrator — it defines a project structure and pipeline API that can be executed by Airflow, Prefect, or directly.

Best for: teams that want opinionated project structure, catalog-based data management, and notebook-to-pipeline promotion Key concepts: DataCatalog (abstracted data sources), nodes (pure functions), pipelines (DAGs of nodes)

Kedro Node and Pipelinepython

# src/fraud_detection/nodes.py
import pandas as pd
from sklearn.ensemble import GradientBoostingClassifier

def featurize(transactions: pd.DataFrame) -> pd.DataFrame:
    return transactions.assign(
        hour=pd.to_datetime(transactions['ts']).dt.hour,
        amount_log=transactions['amount'].apply(lambda x: __import__('math').log1p(x)),
    )

def train_model(features: pd.DataFrame) -> GradientBoostingClassifier:
    X = features.drop("is_fraud", axis=1)
    y = features["is_fraud"]
    return GradientBoostingClassifier(n_estimators=200).fit(X, y)

# src/fraud_detection/pipeline.py
from kedro.pipeline import Pipeline, node, pipeline

def create_pipeline(**kwargs) -> Pipeline:
    return pipeline([
        node(featurize, inputs="raw_transactions", outputs="features"),
        node(train_model, inputs="features", outputs="fraud_model"),
    ])

Choosing the Right Tool

Criteria	DVC	Airflow	Prefect	Kedro
Data versioning	✅	❌	❌	❌
Pipeline execution	✅ (local/CI)	✅ (scheduled)	✅ (event-driven)	✅ (local/Airflow)
Cloud scheduling	❌	✅	✅	Via adapter
Python-native	✅	Partial	✅	✅
Learning curve	Low	High	Medium	Medium
Best fit	Data + model versioning	Enterprise ETL	Modern data teams	DS project structure

In practice, teams often combine tools: DVC for versioning + Airflow or Prefect for scheduling + Kedro for project structure.

Knowledge check

Which of the following does DVC store in Git (not in remote storage)?

Summary

DVC versions datasets and models alongside Git, with remote storage backends and pipeline tracking
DVC pipelines (dvc.yaml) define reproducible, stage-based workflows with automatic caching
Airflow is the industry standard for scheduled DAG orchestration with a rich operator ecosystem
Prefect offers a Python-native alternative with better ergonomics and dynamic workflows
Kedro provides project structure and a DataCatalog abstraction, often paired with Airflow/Prefect
Combine tools: use DVC for versioning, an orchestrator for scheduling, and Kedro for code organization

Next: experiment tracking with MLflow, Weights & Biases, and Neptune.

← MLOps & AI Engineering