GPU engineering

These labs are for the platform team that owns the GPUs: how to share them, tune the operator and check the hardware is performing as it should.

Every lab assumes the environment described in the series overview. Commands and names are examples: replace them with your own.

Labs in this section

• Lab 47: Time-slicing against whole-GPU allocation
• Lab 48: GPU Operator tuning
• Lab 49: MPS as an alternative to time-slicing
• Lab 50: Passthrough health: NUMA and PCIe

Lab 47: Time-slicing against whole-GPU allocation

Goal: Measure what sharing costs and decide where to use it.

Prerequisites

• Labs 3, 10 and 20

Environment notes

Time-slicing gives no memory or fault isolation between pods.

Steps

1. Run a benchmark alone on a whole GPU.
2. Run two and four copies on one time-sliced GPU.
3. Record throughput and latency per copy.
4. Push memory until one copy fails.
5. Write a policy for which workloads may share.

Verification

• You have numbers for 1, 2 and 4 tenants.
• The policy is agreed with users.

Lab 48: GPU Operator tuning

Goal: Manage drivers, the device plugin and node labels with the ClusterPolicy.

Prerequisites

• NVIDIA GPU Operator installed

Environment notes

On FIPS clusters use driver images that support FIPS hosts. Plan driver upgrades like node upgrades.

Steps

1. Read the ClusterPolicy and each operand's status.
2. Change the device plugin config (for example time-slicing replicas) and watch nodes update.
3. Read the GPU feature labels on nodes.
4. Upgrade the driver version on one node pool.
5. Roll back if the smoke test fails.

Verification

• All operands are ready after each change.
• Nodes advertise the expected GPU count.

Lab 49: MPS as an alternative to time-slicing

Goal: Try NVIDIA's Multi-Process Service and compare it with time-slicing.

Prerequisites

• Lab 47

Environment notes

The L40S has no MIG, so MPS and time-slicing are the sharing options. MPS can limit memory per client.

Steps

1. Enable MPS in the device plugin config for a test node.
2. Run the Lab 47 benchmark with two and four clients.
3. Set per-client memory limits.
4. Compare with time-slicing.
5. Return the node to its normal config.

Verification

• You have a comparison table.
• Memory limits stop a client exceeding its share.

Lab 50: Passthrough health: NUMA and PCIe

Goal: Check that GPUs passed through to VMs get the bandwidth and topology they should.

Prerequisites

• Root access to GPU nodes or a privileged debug pod

Environment notes

When GPUs are passed into virtual machines, CPU pinning and NUMA alignment affect performance.

Steps

1. Run nvidia-smi topo -m and read the topology.
2. Run nvbandwidth for host-to-device and device-to-host bandwidth.
3. Run p2pBandwidthLatencyTest between GPUs on one node.
4. Compare with the card's expected PCIe figures.
5. Record the results as a baseline.

Verification

• Bandwidth is close to the PCIe link's expected rate.
• The baseline is stored for future comparison.