Gen AI studio and RAG

Labs 21–27: the Playground, RAG with vector stores, AutoRAG, Llama Stack APIs, MCP tools, guardrails and LLM evaluation.

Updated CompTech Team

Gen AI studio is new in OpenShift AI 3.x and is built on Llama Stack. These labs move from chatting in the Playground to building and evaluating real applications.

Every lab assumes the environment described in the series overview. Commands and names are examples: replace them with your own.

Labs in this section

Lab 21: The Playground

Goal: Use the Playground to compare models, prompts and settings before writing code.

Prerequisites

  • Gen AI studio enabled
  • At least one served LLM (Lab 13)

Environment notes

The Playground uses a Llama Stack server per project. Models must be registered as AI asset endpoints to appear.

Steps

  1. Make a served model available as an AI asset endpoint.
  2. Open Gen AI studio → Playground and pick the project and model.
  3. Change the system prompt, temperature and max tokens.
  4. Compare answers from a 1B and an 8B model.
  5. Export the settings as code.

Verification

  • The Playground returns answers from both models.
  • You can explain how temperature changed the output.

Lab 22: RAG: vector stores, chunking and embeddings

Goal: Build retrieval-augmented generation and understand the choices that matter.

Prerequisites

  • Lab 21
  • An embedding model registered with Llama Stack (Granite embedding or an embedding NIM)

Environment notes

Vector stores can be Milvus Lite inside the Llama Stack pod for labs, or a standalone Milvus or pgvector for anything shared.

Steps

  1. Create a vector store and upload a set of documents.
  2. Ask questions in the Playground with RAG on.
  3. Change chunk size and overlap and re-index.
  4. Swap the embedding model and compare retrieved passages.
  5. Write ten test questions with known answers.

Verification

  • The right passage is retrieved for most test questions.
  • You have a recommended chunk size and embedding model for your documents.

Lab 23: AutoRAG

Goal: Let AutoRAG search RAG settings for you and judge the results.

Prerequisites

  • Lab 22

Environment notes

AutoRAG runs many indexing and retrieval variants, so it uses GPU time; schedule it with Kueue (Lab 32).

Steps

  1. Prepare documents and a question-and-answer test set.
  2. Start an AutoRAG experiment with a small search space.
  3. Read the ranking of configurations and their scores.
  4. Deploy the best configuration.
  5. Check it against your manual result from Lab 22.

Verification

  • The experiment completes and reports scores per configuration.
  • The chosen configuration does at least as well as your manual one.

Lab 24: Llama Stack APIs: agents, tools and the Responses API

Goal: Use Llama Stack from code instead of the Playground.

Prerequisites

  • Labs 21–22
  • A workbench with the llama-stack-client package

Environment notes

Small models are unreliable at tool calling. Use an 8B or larger model for agent labs.

Steps

  1. List models and vector stores with the client.
  2. Send a chat completion and a Responses API request.
  3. Add a file-search tool over your vector store.
  4. Build an agent that uses a tool and keeps conversation state.
  5. Log each step to see what the agent decided.

Verification

  • The agent answers a question that needs the tool, with a citation.
  • You can read the tool calls in the response.

Lab 25: MCP tool servers

Goal: Give a Llama Stack agent tools through the Model Context Protocol.

Prerequisites

  • Lab 24

Environment notes

Run MCP servers inside the cluster and restrict their egress. Treat tool output as untrusted input to the model.

Steps

  1. Deploy a simple MCP server (for example one that reads a test database).
  2. Register it with Llama Stack as a tool group.
  3. Ask the agent a question that needs the tool.
  4. Add a second server and watch tool selection.
  5. Remove the agent's access and confirm it fails safely.

Verification

  • The agent calls the MCP tool and uses the result.
  • Without access, it says it cannot answer rather than inventing data.

Lab 26: Guardrails with TrustyAI

Goal: Filter prompts and answers for PII, prompt injection and toxic content.

Prerequisites

  • Lab 13
  • TrustyAI enabled

Environment notes

The Guardrails orchestrator sits in front of the model and calls detector models, which need their own CPU or GPU resources.

Steps

  1. Deploy the Guardrails orchestrator for a served model.
  2. Add PII, prompt-injection and toxicity detectors.
  3. Send clean and harmful prompts.
  4. Set some detectors to block and some to log only.
  5. Measure the latency added.

Verification

  • Harmful prompts are blocked or flagged.
  • Clean prompts pass with acceptable extra latency.

Lab 27: Evaluating LLMs with LMEval

Goal: Benchmark models with lm-eval-harness through TrustyAI LMEval jobs.

Prerequisites

  • Lab 13
  • TrustyAI enabled

Environment notes

Disconnected sites must mirror the datasets and allow the job to run offline.

Steps

  1. Create an LMEvalJob against a served model with a small task such as a subset of MMLU.
  2. Watch the job pod and read the results.
  3. Repeat for a second model.
  4. Add a custom task built from your own questions.
  5. Record the results beside your benchmark numbers from Lab 20.

Verification

  • Both jobs complete with scores.
  • You can recommend a model on quality and cost together.
← PreviousModel serving with KServeNext →Training and fine-tuning