Count the stands at any IT conference this year that promise an "AI data platform". Dell sells one by that name. HPE sells the same idea as a "unified data layer". NetApp, VAST Data, WEKA and Everpure, the company that was called Pure Storage until February, all have their own. Databricks, Snowflake, Oracle, IBM and Cloudera describe their products in similar terms, and NVIDIA publishes a reference design for the whole category. Red Hat often appears on the list too, although it sells no product by that name.
The rush has a reason. Gartner predicts that through 2026, organisations will abandon 60% of AI projects that are not supported by AI-ready data, and 63% of organisations either lack the right data management practices for AI or are not sure whether they have them. Models have become easy to get. Data that a model can safely use has not.
This post maps the ecosystem as it stands in October 2026: the classic data pipelines that still do most of the work, the three families of AI data platform, the other layers of an enterprise AI stack, and how to choose when the data is not allowed to leave the building.
The map
Each layer depends on the ones beneath it. Infrastructure carries the pipelines, the pipelines fill a lakehouse, an AI data platform turns documents into something a model can search, an AI platform serves the models, a governance layer controls who calls what, and agents and assistants are what people finally see.
Every row has three kinds of answer: open-source projects you run yourself, vendor products you can install in your own data centre, and public cloud services. Most organisations end up mixing columns. The right mix depends less on features than on where the data is allowed to live.
Classic data pipelines still do most of the work
Long before anyone mentions AI, a bank's data already flows through dozens of pipelines: the core banking end-of-day batch, regulatory returns to Bangladesh Bank, reconciliation with card and payment networks, finance and risk reports, and the dashboards management reads every morning. A ministry has the same pattern with files, payments and case records. None of this is AI, and none of it is going away.
.jpg){kind=link}
AI depends on the same plumbing. Data is "AI-ready" when it is clean, deduplicated, current, labelled with who may see it and traceable to its source, and those are exactly the disciplines data engineering teams have practised for twenty years. When an assistant answers from last year's circular, that is not a model problem. It is a pipeline problem.
The money says the same thing. The most telling data deals of the past eighteen months were for classic pipeline companies, bought by vendors whose pitch is AI:
| Deal | What changed hands | Status |
|---|---|---|
| Salesforce and Informatica | Data integration, quality, catalogue and master data management | Completed November 2025, about $8 billion |
| IBM and Confluent | The commercial company behind Apache Kafka, for streaming data | Completed March 2026, about $11 billion |
| Fivetran and dbt Labs | Data loading and SQL transformation | Merger completed June 2026 |
| Databricks and Neon; Snowflake and Crunchy Data | Managed PostgreSQL | Announced May and June 2025 |
What has changed is the shape of the pipeline, not the need for it:
- Documents join the pipeline. PDFs, scans, images and audio are now parsed, split into chunks, embedded and indexed alongside the tables.
- Real time matters more. Agents and fraud models need current data, so change data capture and streaming, typically Kafka with Debezium, are replacing the nightly batch for many feeds.
- The lakehouse is winning. Open table formats such as Apache Iceberg let one copy of the data serve SQL reports, Spark jobs and AI, instead of a warehouse and a data lake kept in step by hand.
- AI helps build pipelines. It writes SQL, maps fields, documents lineage and flags anomalies. Engineers still design and own the result.
- Governance has to reach the answer. Access rules and lineage must follow the data into the vector index, or an assistant will happily quote a document its user was never allowed to open.
What an AI data platform actually does
Strip away the branding and an AI data platform adds five jobs to ordinary storage:
- Find. Scan file shares and object stores, and catalogue what exists, who owns it and who may read it.
- Prepare. Parse documents, images and audio, extract text and tables, and split them into chunks.
- Index. Create vector embeddings and keyword indexes, and keep them in step as files change.
- Serve. Answer queries from RAG pipelines and agents quickly, increasingly through the Model Context Protocol (MCP).
- Govern. Carry permissions, lineage and retention rules from the source file to every answer.
NVIDIA turned this into a category in March 2025 with the NVIDIA AI Data Platform, a reference design for storage systems that run AI query agents on NVIDIA GPUs and networking next to the data. DDN, Dell, HPE, Hitachi Vantara, IBM, NetApp, Nutanix, Pure Storage, VAST Data and WEKA signed up at launch. They make up the first of three families of product.
Family one: storage-led platforms
Storage vendors start where the files already are and add the five jobs on top. The two largest server vendors show the two ways to do it.
Dell AI Data Platform
Dell sells its platform as the data foundation of the Dell AI Factory with NVIDIA, built from two kinds of engine:
- Storage engines. PowerScale for file, ObjectScale for S3-compatible object storage, and the new Lightning File System, a parallel file system for training and inference. Exascale Storage, announced for the second half of 2026, runs file, object, parallel file and block storage on the same PowerEdge servers.
- Data engines. The Data Analytics Engine, powered by Starburst, runs SQL across lakes, databases and object stores without copying the data. The Data Processing Engine is Apache Spark, with NVIDIA cuDF acceleration due in December 2026. The Data Search Engine, powered by Elastic, provides semantic and hybrid search for RAG. The Data Orchestration Engine, built on Dataloop technology that Dell bought in December 2025, ingests, labels and transforms data and keeps embeddings current.
In October 2026 Dell added a roadmap for the first half of 2027: a unified semantic layer, an enterprise knowledge graph and "knowledge agents", which together give AI agents a governed view of company data that respects each user's permissions.
HPE: a "unified data layer"
HPE spreads the same jobs across three products:
- HPE Alletra Storage MP X10000 is object storage that gained native file support in the second quarter of 2026. Optional Data Intelligence Nodes, with NVIDIA GPUs running NVIDIA AI Enterprise, tag incoming data with metadata, create vector embeddings inside the storage, and expose the results to agents through built-in MCP servers.
- HPE Data Fabric Software puts files, objects and databases across sites into one namespace and supports Apache Iceberg. Its May 2026 release added policy-based data placement, metadata classification and lineage, and governance through Apache Polaris.
- HPE AI Essentials, part of HPE Private Cloud AI, supplies the pipeline tools: Airflow, Spark, Ray, the EzPresto SQL engine and Superset dashboards. Private Cloud AI is also offered in an air-gapped version.
Set side by side, the two cover the same ground:
| Job | Dell AI Data Platform | HPE |
|---|---|---|
| File and object storage | PowerScale, ObjectScale, Exascale Storage | Alletra X10000 (object and file), B10000 (block) |
| Prepare and index documents | Data Search Engine, Data Orchestration Engine | X10000 Data Intelligence Nodes |
| SQL across sources | Data Analytics Engine (Starburst) | EzPresto, Data Fabric namespace |
| Processing | Data Processing Engine (Spark) | Spark, Airflow and Ray in AI Essentials |
| Governance and business meaning | Semantic layer and knowledge graph, due 2027 | Data Fabric metadata, lineage and Polaris, available now |
| Packaged AI stack | Dell AI Factory with NVIDIA | HPE Private Cloud AI |
The rest of the field
- NetApp pairs AFX, a disaggregated all-flash system for AI, with the AI Data Engine (AIDE), co-engineered with NVIDIA and launched in October 2025.
- VAST Data calls its product an AI Operating System: storage and a database in one system, with an event engine, an InsightEngine that vectorises data automatically and applies the source files' permissions to search results, and an AgentEngine for running agents.
- Everpure, Pure Storage's name since February 2026, is repositioning from storage to data management, and agreed the same month to buy 1touch for data discovery and orchestration.
- WEKA made its NeuralMesh AI Data Platform, built on the NVIDIA design, generally available in March 2026.
- DDN, IBM, Hitachi Vantara and Nutanix offer high-performance AI storage built around the same NVIDIA design.
Three cautions apply to the whole family. Performance figures are the vendors' own. Some of the most interesting features are still roadmap items, so ask which parts ship today. And most of these designs assume NVIDIA GPUs at every layer, which ties the data platform to one supplier's prices and delivery times.
Family two: software data platforms
The second family started in software and runs on whatever storage it is given. It includes the most widely used data platforms in the world, and for any of them the first question in Bangladesh is where it can run.
| Platform | What it is | Where it runs |
|---|---|---|
| Databricks Data Intelligence Platform | Lakehouse with Unity Catalog governance and Mosaic AI for models and agents | Public cloud |
| Snowflake AI Data Cloud | Warehouse and lakehouse with Cortex AI functions, search and agents | Public cloud; sovereign options through partner-run clouds, mainly in Europe |
| Microsoft Fabric, Google BigQuery, Amazon SageMaker | Each hyperscaler's own data and AI platform | Public cloud |
| Oracle Database and Autonomous AI Lakehouse | Vector search inside the database, and an open Iceberg lakehouse launched in October 2025 | Oracle's cloud, other clouds, or Exadata Cloud@Customer in your data centre; the database itself fully on premises |
| IBM watsonx.data | Open lakehouse on Iceberg, Presto and Spark, now fed live by Confluent | IBM Cloud and AWS, or on premises on Red Hat OpenShift |
| Cloudera | Hybrid data and AI platform, the successor to the Hadoop distributions | Public cloud, or on premises on OpenShift or Cloudera's own container platform, including air-gapped installs |
| Palantir Foundry and AIP | Data integration and an "ontology" of the organisation with AI on top, strong in defence and government | Cloud or customer sites; proprietary and US-controlled |
Where Red Hat fits. Red Hat does not sell an AI data platform. Red Hat AI Enterprise, launched in February 2026, is an AI platform: it brings Red Hat AI Inference Server, OpenShift AI and RHEL AI together on OpenShift, and combined with NVIDIA AI Enterprise it is sold as the Red Hat AI Factory with NVIDIA. Its data pieces sit at the edges, in a hardened package index with Docling for document conversion, SDG Hub for synthetic data and Training Hub for model tuning, alongside Streams for Apache Kafka for streaming. Its larger role in data is underneath: IBM's and Cloudera's on-premises platforms both run on OpenShift.
Family three: open-source building blocks
Underneath most of the products above sit the same open-source projects: Kafka and Debezium for streaming and change data capture; Spark, Airflow and dbt for processing and transformation; Iceberg tables queried with Trino; Milvus, pgvector, OpenSearch or Elasticsearch for vectors; and Docling for documents. Three of Dell's four data engines are Starburst (built on Trino), Spark and Elastic, and HPE's AI Essentials ships Airflow, Spark and a Presto engine.
Assembling these yourself costs engineering time and needs a support plan. In return it runs anywhere, including fully disconnected sites, and keeps every byte in open formats that another platform can read later. For many organisations the sensible answer is open formats at the bottom, and a vendor product wherever one product and one support contract are worth the price.
The rest of the enterprise AI stack
Data is one layer of several. A production AI service needs all of these, and each has its own market:
| Layer | What it does | Popular choices | Runs air-gapped? |
|---|---|---|---|
| Infrastructure | GPU servers, network and storage, sold as tested "AI factory" bundles | NVIDIA DGX, Dell AI Factory, HPE Private Cloud AI, Lenovo, Cisco, Supermicro; GPU clouds such as CoreWeave and Nebius | Yes, the on-premises bundles |
| AI platform | Build, tune, deploy and monitor models | Red Hat OpenShift AI, NVIDIA AI Enterprise, Dataiku, Domino; Microsoft Foundry, Google's Gemini Enterprise Agent Platform (formerly Vertex AI), Amazon Bedrock and SageMaker | Red Hat, NVIDIA, HPE, and self-hosted Dataiku and Domino |
| Inference serving | Run models fast and at a low cost per answer | NVIDIA NIM, Dynamo and Triton; vLLM and llm-d; Red Hat AI Inference Server | Yes |
| AI gateway | One front door to every model: keys, token quotas, cost tracking, logs and routing | WSO2 AI Gateway, Kong AI Gateway, LiteLLM; Google's Agent Gateway | WSO2, Kong and LiteLLM |
| Agent platforms | Agents that take actions in business systems | Salesforce Agentforce, ServiceNow, Microsoft Copilot Studio, IBM watsonx Orchestrate, Amazon Bedrock AgentCore, Gemini Enterprise; LangGraph, CrewAI, LlamaIndex | The open-source frameworks, and the MCP catalog in OpenShift AI |
| Enterprise assistants | Ready-made AI for every employee: chat, documents, email and search | Microsoft 365 Copilot, ChatGPT Enterprise, Claude Enterprise, Gemini Enterprise, Glean | No; build a self-hosted assistant instead |
| Security and governance | Filter prompts and answers, attack-test models, keep an audit trail | Palo Alto Prisma AIRS, Cisco AI Defense, Check Point (which bought Lakera), IBM watsonx.governance; NeMo Guardrails, garak | NeMo Guardrails and garak; others vary |
| Observability and evaluation | Trace every request, its cost and speed, and test the quality of answers | Datadog, Arize, LangSmith, Weights & Biases; Langfuse, MLflow, Arize Phoenix | The open-source tools |
Two of these layers get less attention than they deserve. An AI gateway is how an IT department stays in control once ten teams are calling five models: one place to issue keys, cap token spending and log every call. WSO2 API Manager, for example, now includes an AI gateway with token-based rate limits. Evaluation is what separates a demo from a service: a fixed set of real questions with known answers, run again after every change of model, prompt or document set.
What this means for Bangladesh
For a bank or a government office in Bangladesh, the first filter is not features but location. Defence and many government systems cannot send data out of the building, which rules out the cloud-only platforms whatever their merits. Banks have their own rules on where customer data may be kept, and many run their core banking on Oracle databases, which makes Oracle's options, and Oracle GoldenGate for change data capture, natural starting points.
Six questions sort the options quickly:
- Can it be installed, patched and licensed with no internet connection?
- Does the data stay in open formats, such as Iceberg, Parquet and PostgreSQL, so you can move it later?
- Do access rights follow the data into the vector index and into every answer?
- Which features ship today, and which are on a roadmap?
- Who supports it in Bangladesh, and can your own team run it?
- What does it need from NVIDIA, and what happens to the licence if you change hardware?
The pattern we see working for sovereign deployments is an open-source pipeline on Red Hat OpenShift (Kafka and Debezium, Spark and Airflow, Iceberg with Trino, and a vector database), enterprise storage from Dell, HPE or NetApp underneath, OpenShift AI and NVIDIA AI Enterprise for the models, a gateway such as WSO2 in front, and NeMo Guardrails and audit logging around the whole. A vendor's data engines can replace any layer where a customer prefers one product and one support contract.
Above all, start with the data, not the GPUs:
- List the sources the first use case needs, and who owns each one.
- Fix the pipelines that feed them: freshness, duplicates and access labels.
- Pick one use case with a result you can measure, such as answering staff questions from circulars or flagging suspicious transactions.
- Choose the platform by constraint: where the data may live, the skills you have and the support you can get. Then size the GPUs.
CompTech designs and builds data and AI platforms for banks and government in Bangladesh, on premises and air-gapped where required, with Red Hat, NVIDIA, Dell and HPE. If you are weighing these platforms, talk to us before the first purchase order.
Cover image: a map of the internet's connections by The Opte Project, CC BY 2.5. Photographs are credited in their captions and used under their Creative Commons licences or public-domain dedication from Wikimedia Commons. Diagrams by CompTech. Product names, features and dates are as announced by each vendor up to October 2026; performance figures are vendor claims.
