Pipelines and MLOps

Labs 34–41: pipelines, scheduling and caching, MLflow, the model registry, the model catalog, Feast, GitOps and Tekton CI/CD.

Updated CompTech Team

These labs turn one-off experiments into repeatable processes: train, track, register, deploy and promote, across both sites.

Every lab assumes the environment described in the series overview. Commands and names are examples: replace them with your own.

Labs in this section

Lab 34: Data science pipelines with the KFP SDK

Goal: Build a pipeline that trains, evaluates, registers and deploys a model.

Prerequisites

  • Lab 8
  • A pipeline server configured in the project

Environment notes

The pipeline server needs an S3 bucket for artifacts and a database.

Steps

  1. Write components for data prep, training and evaluation with the KFP SDK.
  2. Compile and upload the pipeline.
  3. Run it from the dashboard.
  4. Add a condition that only registers the model if accuracy passes a threshold.
  5. Add a deploy step.

Verification

  • The run completes and its artifacts are in S3.
  • A failed threshold skips registration.

Lab 35: Scheduled runs and caching

Goal: Run pipelines on a schedule and skip steps whose inputs didn't change.

Prerequisites

  • Lab 34

Environment notes

Caching is keyed on component inputs; disable it for steps that read changing external data.

Steps

  1. Create a recurring run.
  2. Run the pipeline twice and see cached steps.
  3. Disable caching for one step.
  4. Change an input and see which steps rerun.
  5. Pause and resume the schedule.

Verification

  • The second run reuses cached steps.
  • The schedule triggers on time.

Lab 36: Experiment tracking with MLflow

Goal: Log runs, metrics and artifacts and compare experiments.

Prerequisites

  • MLflow enabled
  • Lab 28 or 34

Environment notes

MLflow needs a database and an artifact bucket; on two sites decide which one is primary (Lab 53).

Steps

  1. Point a notebook at the MLflow tracking server.
  2. Log parameters, metrics and the model for several runs.
  3. Compare runs in the MLflow UI.
  4. Log from a pipeline step too.
  5. Register the best model.

Verification

  • All runs appear with metrics and artifacts.
  • You can pick a best run by metric.

Lab 37: The model registry

Goal: Version models with metadata and promote them to serving.

Prerequisites

  • Model registry enabled
  • Lab 34

Environment notes

The registry stores metadata; the weights stay in S3 or OCI.

Steps

  1. Create a registry and grant access.
  2. Register a model and two versions with metadata.
  3. Deploy a version from the registry.
  4. Archive the old version.
  5. Look up which deployment uses which version.

Verification

  • Every deployment traces back to a registered version.
  • Archived versions cannot be deployed by mistake.

Lab 38: The model catalog in AI hub

Goal: Find, review and deploy models from the catalog.

Prerequisites

  • Lab 37

Environment notes

In disconnected sites only mirrored catalog models can be deployed.

Steps

  1. Browse the catalog and filter by task and licence.
  2. Read a model card and its benchmark data.
  3. Register a catalog model.
  4. Deploy it.
  5. Add your own model to an internal catalog source.

Verification

  • The catalog model serves from your cluster.
  • Your internal source appears in the catalog.

Lab 39: Feature store with Feast

Goal: Serve the same features for training and online inference.

Prerequisites

  • Feast enabled
  • Lab 8

Environment notes

The offline store can be files in S3; the online store is usually Redis or PostgreSQL.

Steps

  1. Create a feature store and define entities and feature views.
  2. Materialise features to the online store.
  3. Build a training set from the offline store.
  4. Fetch online features at inference time.
  5. Change a feature and re-materialise.

Verification

  • Training and online features agree for the same entity.
  • The online lookup is fast enough for serving.

Lab 40: GitOps for models across sites

Goal: Deploy model servers to both sites from Git with Argo CD.

Prerequisites

  • OpenShift GitOps installed on both sites
  • Lab 12

Environment notes

Keep site differences (hostnames, storage classes, replica counts) in overlays, not copies.

Steps

  1. Put the InferenceService and runtime in a Git repository with Kustomize overlays per site.
  2. Create an Argo CD application per site.
  3. Change the model version in Git and watch both sites sync.
  4. Make a manual change in the cluster and watch Argo CD revert it.
  5. Roll back with a Git revert.

Verification

  • Both sites run the version in Git.
  • Drift is detected and corrected.

Lab 41: CI/CD with OpenShift Pipelines

Goal: Build, scan, deploy and smoke-test a model-serving image or application.

Prerequisites

  • OpenShift Pipelines installed
  • Lab 40

Environment notes

Image scanning and signing support the PCI evidence in Labs 56–60.

Steps

  1. Write a Tekton pipeline that builds an image with Buildah.
  2. Add a vulnerability scan and fail on critical findings.
  3. Push and sign the image.
  4. Update the GitOps repository so Argo CD deploys it.
  5. Run a smoke test against the endpoint.

Verification

  • A commit produces a deployed, tested image.
  • A critical vulnerability stops the pipeline.
← PreviousTraining and fine-tuningNext →Monitoring and trust