AI data platform
The data layer that assistants and agents run on: document pipelines, vector and keyword indexes, retrieval with access labels, and the storage to feed GPUs, on your own hardware or a vendor's AI data platform.
An assistant is only as good as the data it can reach
Most AI pilots fail on data, not on models. The documents were scanned badly, duplicated across shares, out of date, or not allowed to be shown to the person asking. Gartner expects organisations to abandon most AI projects that lack AI-ready data, and in our experience that is where the work is.
We build the AI data layer as part of the data platform: pipelines that parse, chunk and embed documents from every source, a vector and keyword index that carries each document's access labels, retrieval that returns the right passages with their sources, and the storage and networking to keep GPUs fed. It runs on OpenShift AI and NVIDIA AI Enterprise on your own hardware, or on Dell, HPE and NetApp AI data platforms where you prefer one vendor.
When it fits
01
The pilot answered from the wrong documents
The assistant worked on a sample set, then quoted stale circulars and the wrong version of a policy in production.
02
Access is a blocker
Documents cannot be indexed because nobody can guarantee the assistant will respect who may read what.
03
The archive is paper
Years of records exist as scans and PDFs, in Bangla and English, and none of it can be searched.
04
GPUs are waiting for data
Expensive accelerators sit idle while models load from slow storage and the index rebuilds every night.
What we do
Document pipelines
Parse PDFs, scans, Office files and email with layout-aware tools and OCR, including Bangla, then chunk, deduplicate and embed them, re-running whenever a source changes.
Vector and keyword indexes
Milvus, pgvector, Elasticsearch or OpenSearch, sized for your corpus, with hybrid search and reranking, and each chunk labelled with who may see it.
Retrieval services
Retrieval APIs that return the right passages with their sources, enforce access labels, and feed the assistant and agent applications our AI team builds.
Feature and model data
Governed tables from the lakehouse prepared as features for fraud, risk and forecasting models, with the same lineage as the reports.
Storage for GPUs
NVMe, RDMA networks and GPUDirect-capable storage so models load in seconds and training reads at full speed.
Vendor AI data platforms
Dell AI Data Platform, HPE's unified data layer and NetApp's AI Data Engine, deployed and connected to OpenShift AI where you want one vendor's product.
Retrieval pipeline
Documents in, answers with sources out, access checked on the way
Every document is parsed, chunked and embedded into an index that carries its access labels. A question retrieves only the passages its user may see, and the answer cites the sources it came from. The same pipeline re-runs whenever a document changes, so the assistant is as current as the archive.
- Documents in
- PDFs, scans, Office files and email from shares, archives and systems.
- Prepare and embed
- Layout-aware parsing and OCR, chunking, deduplication and embedding.
- Index with access labels
- Vector and keyword search, with each chunk labelled for who may see it.
- Answers with sources
- Retrieval that returns only permitted passages, cited in the answer.
How an AI data programme runs
01
Choose the corpus
Pick one document set and one use case, such as circulars for staff questions, and agree how the answers will be judged.
02
Design
Design the pipeline, the index, the access model and the GPU and storage sizing, and choose between open source and a vendor platform.
03
Build and evaluate
Build the pipeline and index as code, load the corpus, and measure retrieval and answer quality against a test set before anyone uses it.
04
Extend and operate
Add sources and use cases, keep the index fresh automatically, and run the platform under agreed service levels.
What you get
- Document pipelines for the sources in scope, re-running on change
- A vector and keyword index with access labels, sized for the corpus
- Retrieval APIs with source citations and access enforcement
- A test set and evaluation results for retrieval and answer quality
- GPU storage and networking sized and measured for model loading
- Runbooks, dashboards and training for your team
What we build with
Open-source pipelines and indexes, NVIDIA's packaged microservices, or a vendor's platform, chosen for your case.
Pipelines and models
- Docling
- NVIDIA NeMo Retriever
- NVIDIA RAG Blueprint
- Red Hat OpenShift AI
- vLLM
- NVIDIA NIM
Indexes
- Milvus
- pgvector
- Elasticsearch
- OpenSearch
- Redis
Vendor platforms and storage
- Dell AI Data Platform
- HPE Alletra Storage MP X10000
- NetApp AFX and AI Data Engine
- NVIDIA AI Enterprise
- GPUDirect Storage
Questions about the AI data layer
Does the assistant respect our access rules?
Yes, by design. Each chunk in the index carries the access labels of its source document, and retrieval filters on the user's identity before anything reaches the model. An assistant cannot quote a document its user could not open.
Can it handle Bangla documents?
Yes, with care. Bangla OCR and embeddings are tested early on your own documents, because quality varies between models, and the evaluation set includes Bangla questions from the start.
Open source or a vendor's AI data platform?
Both work, and we build both. The vendor platforms bundle storage, pipelines and search under one contract; the open-source route costs less in licences and keeps the data portable. The engines underneath are largely the same, so the skills carry over. We show you the comparison.
Our map of the AI data platform marketWho builds the assistant itself?
Our AI team, on top of this data layer. The retrieval services are the boundary: the data platform guarantees what can be found and by whom, and the application decides how it is used.
AI and machine learningMake your documents answer questions
Tell us which archive your staff keep asking about. We'll propose a first corpus, the pipeline and index it needs, and how we'll measure the answers.