The enterprise AI stack, layer by layer

Vendors describe their AI products as "platforms", and most of them are, for one layer. A production AI service for a bank or a ministry needs about eight layers working together, and a buyer who knows which layer a product belongs to can compare like with like and spot the gaps.

This article lists the layers from the bottom up, with the products that are popular in each as of October 2026 and whether each can run without an internet connection, which is the first question for most institutions here. The data platform, the layer in the middle, has its own article; the models themselves are covered in our labs.

1. AI infrastructure

What it is: GPU servers, networking and storage, sold as tested bundles that vendors now call "AI factories".

Popular: NVIDIA DGX SuperPOD; Dell AI Factory; HPE Private Cloud AI; Lenovo, Cisco and Supermicro bundles. Or rented from GPU clouds such as CoreWeave and Nebius.

Air-gapped: yes, for all the on-premises bundles.

2. AI platforms

What it is: where teams build, tune, deploy and monitor models.

Popular: Red Hat OpenShift AI and NVIDIA AI Enterprise on premises; Microsoft Foundry (renamed from Azure AI Foundry in January 2026), Google's Gemini Enterprise Agent Platform (which absorbed Vertex AI in April 2026) and Amazon Bedrock and SageMaker in the cloud; Dataiku, Domino and H2O.ai as independents.

Air-gapped: Red Hat, NVIDIA and HPE, plus the self-hosted editions of the independents.

3. Inference serving

What it is: the software that runs models fast and keeps the cost per answer low.

Popular: NVIDIA NIM, Dynamo and Triton; vLLM and llm-d (open source); Red Hat AI Inference Server.

Air-gapped: yes, all of them.

The data platform

Between serving and the layers above sits the data platform: pipelines, databases, the lakehouse, and the document pipelines and vector indexes that feed assistants. It is the layer where most AI projects succeed or fail, and the one our data platform services are built around. The blog post on AI data platforms maps that market.

4. AI gateways

What it is: one controlled front door to every model: keys, token quotas, cost tracking, logging and routing. It is how an IT department stays in control once ten teams call five models.

Popular: Kong AI Gateway, LiteLLM (open source), WSO2's AI Gateway, and cloud gateways such as Google's Agent Gateway. WSO2 API Manager 4.5 and later includes token-based rate limits for AI APIs.

Air-gapped: WSO2, Kong and LiteLLM.

5. Agent platforms

What it is: tools for building agents that take actions in business systems, increasingly connected to those systems through the Model Context Protocol (MCP).

Popular: Salesforce Agentforce, ServiceNow, Microsoft Copilot Studio, IBM watsonx Orchestrate, Amazon Bedrock AgentCore and Gemini Enterprise; the open-source frameworks LangGraph, CrewAI and LlamaIndex.

Air-gapped: the open-source frameworks, and OpenShift AI's MCP catalog, added in version 3.4.

6. Enterprise assistants

What it is: ready-made AI for every employee, covering chat, documents, email and search.

Popular: Microsoft 365 Copilot, ChatGPT Enterprise, Claude Enterprise, Gemini Enterprise, Glean.

Air-gapped: none of these. Air-gapped institutions get a self-hosted assistant built on the layers below, which is what our chatbot lab describes.

7. AI security and governance

What it is: filtering prompts and answers, attack-testing models, and keeping an inventory and audit trail of AI use. The market consolidated through acquisitions: Cisco absorbed Robust Intelligence, Palo Alto Networks folded Protect AI into Prisma AIRS, and Check Point bought Lakera in October 2025.

Popular: Palo Alto Prisma AIRS, Cisco AI Defense, Check Point, IBM watsonx.governance; open source: NVIDIA NeMo Guardrails and the garak vulnerability scanner.

Air-gapped: NeMo Guardrails and garak; the commercial products vary.

8. Observability and evaluation

What it is: tracing every request with its cost and speed, and testing answer quality before and after every change of model, prompt or document set. Evaluation is what separates a demo from a service.

Popular: Datadog LLM Observability, Arize, LangSmith and Weights & Biases; open source and self-hosted: Langfuse, MLflow and Arize Phoenix.

Air-gapped: the open-source tools.

What this means when you buy

• A vendor that sells layers 1 to 3 as one bundle (Dell, HPE, Red Hat with NVIDIA) still leaves you layers 4 to 8 to choose.
• The cloud platforms cover most layers but fail the air-gap test; for a sovereign deployment the realistic stack is open source and the on-premises vendors.
• Two layers are regularly forgotten in proposals: the gateway (control of cost and access) and evaluation (control of quality). Both are cheap compared with the GPUs, and both are what an auditor will ask about.

Our labs cover the platform, serving, agent and guardrail layers today; gateway, security testing and observability topics are planned.