AI and GPU platforms on Kubernetes
Run AI training, fine-tuning and inference on Kubernetes and OpenShift, with GPUs shared fairly between teams, models served in production and data kept where it belongs.
Get more from every GPU
GPU servers are among the most expensive equipment an organisation buys, and too often they sit idle: reserved by one team, waiting for one job, or held by a notebook that uses them once an hour.
Kubernetes turns GPU servers into a shared platform. Jobs queue and run when capacity is free, large GPUs are split for smaller work, and models move from experiment to production through the same pipelines as any other application. We build these platforms on premises, for data that must stay in the country, and in the cloud.
When it fits
01
GPUs are busy one day, idle the next
Capacity is reserved by team rather than shared, so utilisation is low and queues are long at the same time.
02
Models stall before production
Data scientists can train models, but serving them reliably and securely is nobody's job.
03
Data must stay inside
Policies keep sensitive data out of public AI services, so models have to run on infrastructure you control.
04
New GPU servers are arriving
Hardware is bought or planned, and needs a platform before teams can use it.
What we do
GPU nodes on Kubernetes
The NVIDIA GPU Operator installs and updates drivers, device plug-ins and monitoring on every GPU node automatically.
GPU sharing
Multi-Instance GPU splits a large GPU into as many as seven isolated slices, and time-slicing shares GPUs for light work.
Queues and fair shares
Kueue job queues and quotas, so teams share capacity fairly and batch jobs run when GPUs are free.
Model serving
Serve models with KServe and vLLM, with autoscaling, staged rollouts and monitoring of latency and cost.
Notebooks and pipelines
Self-service notebooks, training pipelines and a model registry for data science teams, with OpenShift AI or Kubeflow.
Fast storage and networking
High-throughput storage and networks for training, including GPUDirect where the hardware supports it.
GPU sharing
Share GPUs instead of reserving them
Jobs from every team wait in queues. The scheduler places each one on a whole GPU or a slice of one as capacity frees up, so expensive hardware stays busy and urgent work runs first.
- Training jobs
- Long runs that take whole GPUs, often several at once.
- Inference services
- Models serving users, kept running and scaled with demand.
- Notebooks and experiments
- Small, bursty work that fits on a slice of a GPU.
- GPU utilisation
- Rises as work is packed onto the capacity you already own.
How we build an AI platform
01
Assess
Understand your models, data, users and hardware, and how GPUs are used and shared today.
02
Design
Design GPU node pools, sharing, queues, storage and networking, and choose the serving and data science tools.
03
Build
Build the platform, take a first team and a first model to production, and measure utilisation.
04
Scale
Onboard more teams and models, tune quotas and sharing, and plan capacity from real usage.
What you get
- GPU nodes managed by the NVIDIA GPU Operator
- GPU sharing with MIG and time-slicing, set up for your workloads
- Queues and quotas for each team
- Model serving with autoscaling and monitoring
- Notebooks, pipelines and a model registry
- Dashboards for GPU utilisation and cost
Platforms and tools
- NVIDIA GPU Operator
- NVIDIA AI Enterprise
- Multi-Instance GPU
- Red Hat OpenShift AI
- Kubeflow
- KServe
- vLLM
- NVIDIA Triton Inference Server
- Kueue
- Ray
- MLflow
- NVIDIA DCGM
- GPUDirect Storage
- Prometheus and Grafana
Questions about AI platforms
Which GPUs do you support?
NVIDIA data centre GPUs supported by the GPU Operator, in servers from Dell, HPE and other vendors, on premises or in the cloud. Splitting a GPU into slices needs one that supports MIG, such as the A100, H100 or H200.
Can we run large language models on our own servers?
Yes. Open models can be served with vLLM or NVIDIA's inference software on your own GPUs, so prompts and data stay inside your network. We size the hardware to the models and the number of users.
AI and ML solutionsDo we need OpenShift AI?
Not always. It adds a supported, integrated set of data science tools on OpenShift. Teams on upstream Kubernetes can use Kubeflow, KServe and Kueue directly. We recommend based on your support needs and skills.
How do we stop one team taking every GPU?
With queues and quotas. Each team gets a guaranteed share and can borrow idle capacity, which goes back to its owner when needed.
Where does storage fit in?
Training reads large datasets quickly, so storage often decides how busy the GPUs can be. We design storage and networking with the platform, using GPUDirect Storage where it helps.
GPUDirect Storage explainedPut your GPUs to work
Tell us about your models, your teams and the hardware you have or plan to buy. We'll suggest a platform design and a first use case to prove it.