Platform and administration

Labs 1–6: the DataScienceCluster, dashboard configuration and identity, hardware profiles, custom workbench images, multi-tenancy and operator upgrades.

Updated CompTech Team

Before anyone opens a notebook, someone has to decide which components run, who is an administrator, which hardware a user may ask for and how teams are kept apart. These six labs are that someone's job.

Every lab assumes the environment described in the series overview. Commands and names are examples: replace them with your own.

Labs in this section

Lab 1: DataScienceCluster and DSCInitialization

Goal: See how the OpenShift AI operator turns two custom resources into a running platform, and practise turning components on and off safely.

Prerequisites

  • cluster-admin on a cluster with OpenShift AI 3.x installed
  • The oc CLI

Environment notes

OpenShift AI installs one DSCInitialization (platform-wide settings such as the applications namespace and monitoring) and one DataScienceCluster (which components are Managed or Removed). On a FIPS cluster, check each component's FIPS status before enabling it.

Steps

  1. Read both resources: oc get dsci,dsc -o yaml and note each component's managementState.
  2. List what the operator created in the applications namespace and match deployments to components.
  3. Set one optional component (for example TrustyAI) to Removed, watch the operator delete its workloads, then set it back to Managed.
  4. Edit a managed deployment by hand and watch the operator revert it: this is why changes go through the DSC.
  5. Read the DSC status.conditions and the operator logs during a reconcile.

Verification

  • oc get dsc reports Ready, and every component you set to Managed shows a ready condition.
  • You can explain which settings belong in the DSCI and which in the DSC.

Lab 2: Dashboard configuration and identity-provider groups

Goal: Control dashboard features and decide who is an OpenShift AI administrator or user, using groups from your identity provider.

Prerequisites

  • Lab 1
  • An OpenShift OAuth identity provider backed by Keycloak (or another OIDC IdP) with group sync

Environment notes

In 3.x the dashboard is configured through OdhDashboardConfig and the auth resource in the applications namespace. Groups come from the IdP, so a user's rights change when their group membership changes, not when someone edits a role binding.

Steps

  1. Create ai-admins and ai-users groups in the IdP and sync them into OpenShift.
  2. Point the OpenShift AI admin and allowed-user groups at them.
  3. Toggle dashboard feature flags (for example the model registry or Gen AI studio pages) in OdhDashboardConfig.
  4. Sign in as a member of each group and compare the menus.
  5. Remove a user from ai-users and confirm they lose access after the next sign-in.

Verification

  • An ai-admins member sees Settings; an ai-users member does not.
  • A user outside both groups cannot open the dashboard.

Lab 3: Hardware profiles: CPU, full L40S and time-sliced GPU

Goal: Offer users a short menu of sized environments instead of free-form resource requests.

Prerequisites

  • Lab 1
  • NVIDIA GPU Operator with GPU nodes labelled by Node Feature Discovery

Environment notes

The lab's GPU workers carry NVIDIA L40S cards in PCIe passthrough. A time-slicing config in the GPU Operator can advertise each card as several nvidia.com/gpu resources; label or taint those nodes so profiles can target them.

Steps

  1. Create a CPU-only profile with sensible default and maximum CPU and memory.
  2. Create a full-GPU profile requesting nvidia.com/gpu: 1 with a node selector and toleration for whole-GPU nodes.
  3. Create a time-sliced profile that targets nodes advertising shared replicas.
  4. Mark one profile as the default and hide old accelerator profiles.
  5. Start a workbench with each profile and inspect the pod's resources, node and tolerations.

Verification

  • Each workbench lands on the expected node type with the expected limits.
  • A user cannot request more than the profile maximum.

Lab 4: Custom workbench images

Goal: Build, import and pin a workbench image so every data scientist starts from the same, reproducible environment.

Prerequisites

  • Labs 1–3
  • OpenShift Pipelines or another image build tool
  • A registry the cluster can pull from

Environment notes

Start from a supported OpenShift AI workbench image so FIPS-relevant libraries and the notebook server stay consistent. In a disconnected site the image must be mirrored first (see Lab 58).

Steps

  1. Write a Containerfile FROM a supported Jupyter image and add pinned packages (requirements.txt with exact versions).
  2. Build it with a Tekton pipeline or Buildah and push it to the internal registry.
  3. Import it in Settings → Workbench images with a name, description and package list.
  4. Start a workbench from it and confirm the pinned versions.
  5. Publish a second tag and practise rolling users forward.

Verification

  • pip list in the workbench matches your pinned versions.
  • The image appears to ordinary users in the workbench image list.

Lab 5: Multi-tenancy: projects, RBAC, quotas and limits

Goal: Give each team its own space and make sure one team cannot starve or see another.

Prerequisites

  • Labs 2–3
  • Two test users in different IdP groups

Environment notes

Data science projects are OpenShift namespaces with a dashboard label. GPU quota uses the requests.nvidia.com/gpu resource name. Kueue (Lab 32) adds queue-based fair sharing on top of quotas.

Steps

  1. Create a project per team and grant the team group admin or edit.
  2. Apply a ResourceQuota with CPU, memory, storage and requests.nvidia.com/gpu limits.
  3. Apply a LimitRange with default requests so unsized pods are still counted.
  4. As team A, try to list team B's workbenches and models.
  5. Exceed the GPU quota on purpose and read the error.

Verification

  • Cross-team access is denied.
  • A pod that would exceed quota is rejected with a clear quota message.

Lab 6: Operator upgrade and rollback

Goal: Practise upgrading OpenShift AI with a tested plan, a recovery path and minimal user impact.

Prerequisites

  • Labs 1–5
  • Two sites, or a non-production cluster to rehearse on
  • Backups of project namespaces (Lab 54)

Environment notes

Rehearse on the DR site first, then upgrade DC. Use a Manual install plan approval so upgrades only happen when you approve them. OLM does not support downgrading an operator in place, so “rollback” means restoring from backup or reinstalling a previous version.

Steps

  1. Record the current channel, CSV version, DSC and running workloads.
  2. Read the release notes for removed or renamed components.
  3. Approve the install plan on the rehearsal site and watch the CSV and DSC conditions.
  4. Run a smoke test: a workbench starts, a model answers, a pipeline runs.
  5. Write down what a rollback would take, and test restoring one project from backup.

Verification

  • The new CSV is Succeeded and the DSC is Ready.
  • The smoke test passes on the rehearsal site before you touch production.
← PreviousOverviewNext →Workbenches and data