Monitoring and trust

Labs 42–46: Observe & monitor, GPU telemetry, TrustyAI bias and drift, tracing through the mesh, and logs with an audit trail.

Updated CompTech Team

A model that answers is not the same as a model that is healthy, fair and accountable. These labs add the visibility you need to run AI in production.

Every lab assumes the environment described in the series overview. Commands and names are examples: replace them with your own.

Labs in this section

Lab 42: Observe & monitor and Perses dashboards

Goal: Use OpenShift AI's monitoring for model metrics, dashboards and alerts.

Prerequisites

  • Monitoring enabled in the DSCI
  • Lab 12

Environment notes

Observe & monitor relies on the monitoring stack the operator deploys; the cluster's own Observe menu covers infrastructure.

Steps

  1. Open Observe & monitor and select a deployed model.
  2. Read request rate, latency and error metrics.
  3. Create a Perses dashboard for your models.
  4. Add an alert on error rate.
  5. Trigger it with bad requests.

Verification

  • Model metrics appear for the deployed model.
  • The alert fires and resolves.

Lab 43: GPU telemetry with DCGM

Goal: Watch GPU utilisation, memory, temperature and errors.

Prerequisites

  • NVIDIA GPU Operator with DCGM exporter

Environment notes

The console has an NVIDIA GPU dashboard when the dashboard plugin is installed.

Steps

  1. Query DCGM_FI_DEV_GPU_UTIL and memory metrics in Observe → Metrics.
  2. Open the GPU dashboard.
  3. Run a load and watch the metrics.
  4. Create alerts for high memory and temperature.
  5. Find which pod uses which GPU.

Verification

  • You can see per-GPU and per-pod use.
  • Alerts fire under load.

Lab 44: Bias and drift with TrustyAI

Goal: Monitor a classic model for fairness and data drift.

Prerequisites

  • TrustyAI enabled
  • A tabular model served with KServe

Environment notes

TrustyAI needs inference data logged to its service; plan storage for it.

Steps

  1. Enable TrustyAI for the project.
  2. Send training-like data to the model.
  3. Configure a bias metric (for example statistical parity) on a protected attribute.
  4. Configure a drift metric.
  5. Send skewed data and watch the metrics.

Verification

  • Metrics appear in the dashboard.
  • Skewed data moves the metrics past your thresholds.

Lab 45: Tracing inference with OpenTelemetry and Tempo

Goal: Follow a request from the client through the mesh into the model server.

Prerequisites

  • Service Mesh 3, OpenTelemetry and Tempo installed
  • Lab 12

Environment notes

vLLM and NIM can export OTLP traces; the mesh adds spans for each hop.

Steps

  1. Add the model namespace to the mesh.
  2. Point the mesh and the model server at the OpenTelemetry collector.
  3. Send requests through an application.
  4. Find the trace in Observe → Traces.
  5. Find the slowest span.

Verification

  • One trace shows client, gateway and model spans.
  • You can name the slowest hop.

Lab 46: Logs and an audit trail with Loki

Goal: Find model and workbench logs, and show who deployed what.

Prerequisites

  • Logging (Loki) installed
  • API audit logs forwarded

Environment notes

Audit logs are sensitive; restrict who can read them.

Steps

  1. Query model server logs in Observe → Logs.
  2. Query workbench logs for one user.
  3. Find the audit event for an InferenceService creation.
  4. Build a saved query for deployments by user.
  5. Set retention to meet your policy.

Verification

  • You can answer “who deployed this model and when” from logs.
  • Retention matches policy.
← PreviousPipelines and MLOpsNext →GPU engineering