Data platform architecture and design
A reference architecture sized for your data, with two sites, security and governance designed in, and the vendor choices to match, before anything is bought or built.
A platform drawn for ten years, not for the first project
A data platform carries every report, model and assistant the institution builds on it, and it has to keep working through hardware refreshes, new regulations and the next five AI products. The design decides whether that is routine or a rebuild: where each kind of data lives, how it moves between sites, who may see what, and which parts a vendor owns.
We design the platform as layers with open interfaces between them: infrastructure and storage, the data services, the governance that spans them, and the second site that mirrors them. Each layer can be sized, replaced and upgraded on its own, and the design doubles as the specification you procure against.
When it fits
01
An assessment is done
You know the use cases and the route, and now need the detailed design that procurement and the build team can work from.
02
Procurement is starting
A tender for storage, servers or a data platform is being written, and the specification has to come from a design, not a brochure.
03
Two sites are required
The regulator or your own policy requires a disaster recovery site, and the data platform has to be designed for it from the start.
04
AI has to fit the existing estate
GPUs, vector indexes and document pipelines have to sit beside the warehouse and databases you already run.
What we do
Reference architecture
Draw the layers, the components in each and the interfaces between them, for ingestion, storage, lakehouse, AI data and consumption.
Sizing
Size storage, compute, GPUs, network and licences from your data volumes and growth, with headroom for the roadmap.
DC and DR design
Design replication, recovery objectives and the failover and failback procedures for every data service, and the network between the sites.
Security and governance design
Design identity, access control, encryption, audit, lineage and retention so they apply across the lakehouse, the databases and the vector index alike.
Vendor selection
Compare Dell, HPE, NetApp, Oracle and open-source options against the design, and write the specification for the tender.
Air-gapped design
Design the mirrors, update paths and operating procedures for a platform with no connection to the internet.
Reference architecture
Layers that can be sized, replaced and mirrored on their own
The design separates the infrastructure and storage from the data services on top, runs governance and security across every layer, and mirrors what matters to a second site. Each layer has open interfaces, so a storage refresh or a new engine does not mean a new platform.
- Infrastructure and storage
- Servers, GPUs, network, and file, object and block storage, on premises.
- Data services
- Streaming, databases and caches, the lakehouse, and the AI data layer, on OpenShift.
- Governance and security
- Identity, access control, encryption, lineage, audit and retention, applied to every layer.
- Second site
- The services that must survive a site loss, replicated and rehearsed.
How a design engagement runs
01
Requirements
Confirm the use cases, data volumes, growth, recovery objectives, classifications and the rules the platform must meet.
02
Architecture
Draw the layers and components, decide what runs where, and design the data flows between sources, sites and consumers.
03
Size and select
Size every layer, compare the vendor options against the design, and write the specification.
04
Review and hand over
Walk the design through with your architects, security and operations teams, and hand over the documents the build will follow.
What you get
- A reference architecture with every layer, component and interface drawn
- Data flow designs from each source to each consumer, across both sites
- Sizing for storage, compute, GPUs, network and licences
- A DC and DR design with recovery objectives and failover procedures
- A security and governance design covering identity, access, encryption, lineage and audit
- A vendor comparison and the specification for procurement
- Build and acceptance criteria for the implementation
What we design with
The components change by route; the layers and the interfaces between them stay the same.
Infrastructure and storage
- Dell PowerEdge, PowerScale and ObjectScale
- HPE ProLiant and Alletra Storage
- NetApp ONTAP and StorageGRID
- NVIDIA GPUs and networking
- Oracle Exadata
Data services
- Red Hat OpenShift and OpenShift AI
- Kafka and Debezium
- Oracle and PostgreSQL
- Redis and Data Grid
- Iceberg, Trino and Spark
- Milvus and Elasticsearch
Governance and security
- Keycloak
- OpenMetadata and OpenLineage
- Apache Polaris
- HashiCorp Vault
- Red Hat Advanced Cluster Security
Questions about platform design
Can we reuse storage and servers we already own?
Often, yes. We check what you have against the design and reuse what fits. The sizing shows what, if anything, you need to buy, and the specification is written so existing kit counts.
How do you design for air-gapped sites?
Every component is chosen for an offline install: container images, Helm charts and operators are mirrored inside, updates arrive through a controlled transfer, and no service depends on a cloud endpoint. Our lab runs the same way, so the design is tested before it reaches you.
Does the design lock us into a vendor?
No. The layers have open interfaces, data lives in open formats, and the governance layer is independent of the storage and the engines. A vendor can own a layer without owning the platform.
What happens after the design?
The design is the specification for procurement and the plan for the build. Many clients ask us to build the platform from it, which we run as a separate project.
Streaming and integrationDesign a platform you can procure against
Tell us the use cases, the data volumes and the rules. We'll propose the scope of a design engagement and what you will have at the end of it.