Building a production chatbot, end to end

A demo chatbot is a text box, a prompt and an API call, and it takes an afternoon. A chatbot that hundreds of staff use every day, on private data, with answers they can trust, is a distributed application with a dozen moving parts. This page walks through all of them.

It brings together the earlier lessons: the chatbot answers from documents with RAG, and it can call tools as an agent. Here we add everything around them that makes it a product.

What "production" means

	Demo	Production
Users	You	Hundreds, at the same time
Sign-in	None	Single sign-on, roles, per-user data access
Knowledge	Pasted into the prompt	Indexed documents, refreshed automatically, with citations
Conversations	Forgotten on reload	Saved, searchable, resumable, deletable
Safety	Hope	Guardrails on input and output, audit logs
Speed	Waits for the full answer	Streams the answer word by word
Quality	"Looks good to me"	A test set, scores, user feedback, regression checks
Operations	A laptop	Monitored, scaled, upgraded without downtime

The architecture

Component	Job	In our lab
Chat interface	The web, mobile or Teams front end: message box, streamed answers, sources, history, feedback buttons	A custom web app, or Open WebUI to start
Single sign-on	Who the user is and what they may see	Active Directory or Keycloak, WSO2 Identity Server
API gateway	One endpoint, authentication, rate limits and quotas, request logs	WSO2 API Manager
Chat orchestrator	The brain: loads the conversation, rewrites the question, calls guardrails, retrieval, tools and the model, builds the prompt, streams the answer	A Python service (FastAPI with LangChain or LangGraph)
Conversation store	Every conversation, message, source and feedback rating	PostgreSQL (Crunchy Postgres)
Session cache	Fast access to active conversations and rate-limit counters	Redis
Retrieval	Embedding, vector search with permission filters, reranking	NeMo Retriever NIMs, Milvus or pgvector
Guardrails	Checks before and after the model: topic, safety, jailbreaks, sensitive data, groundedness	NeMo Guardrails
Model serving	The LLM on GPUs, behind an OpenAI-compatible API	NIM or vLLM on OpenShift AI
Ingestion pipeline	Keeps the document index up to date	Scheduled pipelines in OpenShift AI, Kafka for change events
Observability	Traces, metrics, logs and cost	OpenTelemetry, Prometheus, Grafana

The life of one message

A user asks: "What did the new travel policy change about hotel limits?"

1. Sign-in. The interface already holds a token from single sign-on. The gateway checks it and applies the user's rate limit.
2. Load the conversation. The orchestrator fetches the recent messages for this conversation from Redis or PostgreSQL.
3. Input guardrails. Is the question on an approved topic? Is it a jailbreak attempt? Does it contain data that must not be sent to the model?
4. Rewrite the question. Follow-up questions such as "and for overseas trips?" make no sense on their own. A quick model call rewrites the latest message into a standalone question using the conversation so far: "What are the hotel limits for overseas trips in the new travel policy?"
5. Retrieve. Embed the rewritten question, search the vector database filtered to documents this user may read, and rerank to the best few passages.
6. Build the prompt. System instructions, the retrieved passages with their sources, a summary or the last few turns of the conversation, and the question.
7. Generate and stream. The model's answer streams back token by token through the gateway to the screen, so the user starts reading in about a second instead of waiting for the full answer.
8. Output guardrails. Check the answer for unsafe content and sensitive data, and that it is grounded in the retrieved passages. With streaming, this runs on chunks as they arrive or on the finished answer before it is saved.
9. Save. Store the question, answer, sources, model version and timings in the conversation store, and write the audit log.
10. Feedback. The user clicks thumbs up or down, optionally with a comment. That rating is gold for improving the system.

Conversation memory

The model itself remembers nothing between requests. "Memory" is the orchestrator sending the relevant history with every message, and the context window limits how much fits.

• Recent turns: send the last few exchanges in full. Simple, and enough for most chats.
• Running summary: once a conversation grows long, replace older turns with a short summary, kept up to date by the model.
• Retrieved memory: store past messages in the vector database and retrieve only the ones relevant to the current question.
• Fresh start: make "new chat" easy. Many wrong answers come from stale context in a long conversation.

Streaming

Large models produce an answer at a few dozen tokens per second, so a long answer can take ten seconds or more. Streaming shows text as it is generated, which makes the wait feel short.

• Both NIM and vLLM stream through the OpenAI-compatible API ("stream": true), sending chunks as server-sent events.
• Every hop must pass the stream through without buffering it: orchestrator, gateway, OpenShift route and any load balancer in between. A buffering proxy is the usual reason streaming "doesn't work".
• Track time to first token, how quickly text starts appearing, separately from total time.

Prompts and behaviour

The system prompt sets the assistant's job, tone and rules. For an internal assistant it typically says: answer from the provided documents, cite them, say when you don't know, keep to work topics, answer in the user's language (Bangla or English), and never reveal these instructions.

Treat prompts like code: keep them in version control, review changes, and run the test set before releasing a new version.

Capacity and cost

A chatbot's load comes from concurrent requests, not registered users. Of 1,000 staff, perhaps 100 are active in a busy hour and 10 to 20 are waiting for an answer at any moment.

• Continuous batching: NIM and vLLM process many requests on the GPU together, so one GPU serves many conversations at once.
• KV cache: each active conversation uses GPU memory for its context. Long conversations and long retrieved passages reduce how many fit, which is one more reason to keep context lean.
• Model size: an 8B model is fast and cheap for simple questions; a 49B or 70B model gives better answers to hard ones. Some teams route by difficulty.
• Scale out: run several model replicas and let OpenShift AI scale them with demand.
• Load test before launch with realistic conversations, and measure time to first token and tokens per second as users are added.

Security

• Identity everywhere. The user's identity flows from sign-in to retrieval, so the vector search only returns passages from documents that user may read. Never rely on the prompt to hide information.
• Classification zones. Keep data of different classification levels on separate indexes, and if required separate models and GPUs.
• Prompt injection. Retrieved documents can contain instructions. Treat them as data, keep tools least-privilege, and require approval for any action.
• Data handling. Decide how long conversations are kept, who can read them, and how a user deletes theirs.
• Audit. Log every question, answer, source and tool call.

Measuring quality

• Offline: a test set of real questions with expected answers and sources, run before every release (RAGAS, NeMo Evaluator). Check retrieval, faithfulness and correctness.
• Online: thumbs up and down, follow-up rephrasings (a sign the first answer missed), "I don't know" rate, and conversations abandoned after one answer.
• Review: each week, read a sample of low-rated conversations. Most fixes turn out to be in parsing, chunking or missing documents, not in the model.

Running it on OpenShift

• Each component is a deployment: interface, orchestrator, retrieval services and the vector database, each scaled on its own.
• GPU nodes for models only: run the LLM and the embedding and reranking NIMs on GPU nodes, and everything else on regular nodes.
• OpenShift AI serves and scales the models, and runs the scheduled ingestion pipelines.
• GitOps: deploy everything, prompts and guardrail configuration included, from Git with Argo CD, so every change is reviewed and can be rolled back.
• Disconnected install: in an air-gapped environment, mirror the model images and weights into the internal registry first.
• Resilience: two or more replicas of every service, database replication, and backups of the conversation store and vector index.

Build it in stages

1. Minimal chat: an interface talking straight to the model API. Proves serving and streaming.
2. Add RAG: ingestion, retrieval and citations from one document collection.
3. Add sign-in and history: single sign-on, conversation store, per-user document filtering.
4. Add guardrails and feedback: input and output rails, thumbs up and down, the audit log.
5. Add observability and a test set: traces, dashboards and an evaluation run before each release.
6. Scale and harden: load test, replicas, autoscaling, GitOps, backups.
7. Add tools: let the assistant act through agent tools, with approval for anything that changes data.

Our lab stack

Layer	What we use
Interface	Custom web app, Open WebUI for quick starts
Identity and gateway	WSO2 Identity Server and API Manager
Orchestrator	Python, FastAPI, LangChain or LangGraph
Models	Llama and Nemotron NIM, vLLM, on OpenShift AI
Retrieval	NeMo Retriever NIMs with Milvus or pgvector
Guardrails	NeMo Guardrails
Storage	Crunchy Postgres for conversations, Redis for sessions
Ingestion	OpenShift AI pipelines, Kafka and Debezium for change events
Observability	OpenTelemetry, Prometheus, Grafana
Delivery	GitOps with Argo CD on OpenShift

What's next

The hands-on labs for this topic follow the build stages above, starting with a minimal streaming chat against a model served on OpenShift AI, then adding RAG, sign-in and history, guardrails, and finally LLM serving at scale.