OpenShift AI labs: the complete series

OpenShift AI 3.x looks and works differently from the Open Data Hub releases many teams started with: a redesigned dashboard, Gen AI studio built on Llama Stack, KServe as the only serving mode, and new pieces such as llm-d, Kueue and the model catalog. This series covers the whole platform in sixty short labs, grouped into ten sections. Each lab has a goal, prerequisites, notes on the environment, a step outline and a way to check you succeeded.

The lab environment

The labs were written for a typical regulated-sector setup. You don't need all of it to start, and each lab lists what it needs.

Part	In the lab
Platform	Red Hat OpenShift 4.22, installed in FIPS mode
AI platform	Red Hat OpenShift AI 3.x
GPUs	NVIDIA L40S (48 GB, PCIe, no NVLink or MIG), passed through to GPU worker nodes, managed by the NVIDIA GPU Operator, with optional time-slicing
Models	NVIDIA NIM microservices, vLLM and OpenVINO runtimes
Storage	NetApp ONTAP through Trident (NAS and SAN), and OpenShift Data Foundation (Ceph and the Multicloud Object Gateway for S3)
Identity	Keycloak as the OpenID Connect provider, with groups
Networking and observability	OpenShift Service Mesh 3, OpenTelemetry, Tempo, Loki and the Cluster Observability Operator
Sites	Two clusters, a primary data centre (DC) and a disaster-recovery site (DR)

Names, hostnames and addresses in the labs are examples. Keep tokens such as NGC and Hugging Face keys in secrets, never in notebooks or Git.

The sections

Section	Labs	What you'll learn
Platform and administration	1–6	the DataScienceCluster, dashboard configuration and identity, hardware profiles, custom workbench images, multi-tenancy and operator upgrades.
Workbenches and data	7–11	workbench types with GPU profiles, S3 data connections, workbench storage, a GPU sanity check and Git with Elyra.
Model serving with KServe	12–20	OpenVINO and vLLM, NVIDIA NIM, custom runtimes, OCI modelcars, secure endpoints, autoscaling, rollouts, llm-d and benchmarking.
Gen AI studio and RAG	21–27	the Playground, RAG with vector stores, AutoRAG, Llama Stack APIs, MCP tools, guardrails and LLM evaluation.
Training and fine-tuning	28–33	single-GPU fine-tuning, Kubeflow Trainer, LoRA and QLoRA, Ray, Kueue and checkpointing.
Pipelines and MLOps	34–41	pipelines, scheduling and caching, MLflow, the model registry, the model catalog, Feast, GitOps and Tekton CI/CD.
Monitoring and trust	42–46	Observe & monitor, GPU telemetry, TrustyAI bias and drift, tracing through the mesh, and logs with an audit trail.
GPU engineering	47–50	time-slicing against whole GPUs, GPU Operator tuning, MPS, and passthrough health.
DC/DR and resilience	51–55	model failover, replicated model storage, platform failover, backup and restore, and multi-cluster networking.
Security and compliance	56–60	FIPS posture, network policies and egress, disconnected operation, secrets and Compliance Operator scans.

All sixty labs

Platform and administration

1. DataScienceCluster and DSCInitialization
2. Dashboard configuration and identity-provider groups
3. Hardware profiles: CPU, full L40S and time-sliced GPU
4. Custom workbench images
5. Multi-tenancy: projects, RBAC, quotas and limits
6. Operator upgrade and rollback

Workbenches and data

7. Jupyter, VS Code and RStudio workbenches
8. S3 data connections with NooBaa and ONTAP S3
9. Workbench storage: ontap-nas, ontap-san, ODF and RWX
10. GPU sanity check in a notebook
11. Git integration and Elyra pipelines

Model serving with KServe

12. OpenVINO Model Server and vLLM, CPU against GPU
13. NVIDIA NIM: an 8B LLM and an embedding NIM
14. Custom vLLM runtime with Hugging Face models and quantization
15. Modelcar: serving a model from an OCI image
16. Secure endpoints: authentication, routes and rate limits
17. Autoscaling with HPA and KEDA, including scale to zero
18. Canary and A/B rollouts between model versions
19. Distributed inference with llm-d
20. Benchmarking with GuideLLM and vLLM bench

Gen AI studio and RAG

21. The Playground
22. RAG: vector stores, chunking and embeddings
23. AutoRAG
24. Llama Stack APIs: agents, tools and the Responses API
25. MCP tool servers
26. Guardrails with TrustyAI
27. Evaluating LLMs with LMEval

Training and fine-tuning

28. Single-GPU fine-tuning in a workbench
29. Distributed training with Kubeflow Trainer
30. LoRA and QLoRA fine-tuning
31. Ray and CodeFlare: Ray Train and Ray Tune
32. Kueue: quotas, fair share, preemption and borrowing
33. Checkpointing to ONTAP and ODF

Pipelines and MLOps

34. Data science pipelines with the KFP SDK
35. Scheduled runs and caching
36. Experiment tracking with MLflow
37. The model registry
38. The model catalog in AI hub
39. Feature store with Feast
40. GitOps for models across sites
41. CI/CD with OpenShift Pipelines

Monitoring and trust

42. Observe & monitor and Perses dashboards
43. GPU telemetry with DCGM
44. Bias and drift with TrustyAI
45. Tracing inference with OpenTelemetry and Tempo
46. Logs and an audit trail with Loki

GPU engineering

47. Time-slicing against whole-GPU allocation
48. GPU Operator tuning
49. MPS as an alternative to time-slicing
50. Passthrough health: NUMA and PCIe

DC/DR and resilience

51. Model failover and RTO
52. Replicating model storage with SnapMirror
53. Platform failover: identity, registry and MLflow
54. Backup and restore with OADP
55. Multi-cluster mesh and Submariner

Security and compliance

56. FIPS posture and the NIM exception
57. NetworkPolicies and egress control
58. Disconnected operation with mirrored images and models
59. Secrets with External Secrets or Vault
60. Compliance Operator scans for PCI DSS and CIS

Suggested order

1. Labs 1–15: set up the platform, workbenches and data, and serve your first models.
2. Labs 21–27: Gen AI studio, RAG, agents, guardrails and evaluation.
3. Labs 28–41: training, fine-tuning, pipelines and MLOps.
4. Labs 51–60: resilience across sites, then security and compliance.

Fit the scaling and benchmarking labs (16–20), monitoring (42–46) and GPU engineering (47–50) in as you need them; they are most useful once real workloads are running.