Training and fine-tuning

Labs 28–33: single-GPU fine-tuning, Kubeflow Trainer, LoRA and QLoRA, Ray, Kueue and checkpointing.

Updated CompTech Team

Most organisations fine-tune rather than train from scratch. These labs cover the tools, queueing GPU jobs fairly, and not losing a day of work to a killed pod.

Every lab assumes the environment described in the series overview. Commands and names are examples: replace them with your own.

Labs in this section

Lab 28: Single-GPU fine-tuning in a workbench

Goal: Fine-tune a small model on one GPU and save the result for serving.

Prerequisites

  • Labs 3 and 10
  • A small dataset in S3

Environment notes

Use the full-GPU profile; time-sliced GPUs share memory and may run out mid-job.

Steps

  1. Load a small model and dataset in a PyTorch workbench.
  2. Fine-tune for a short run with Hugging Face Trainer.
  3. Watch GPU memory and utilisation.
  4. Save the model to S3.
  5. Serve it (Lab 14) and compare with the base model.

Verification

  • Training loss falls and the saved model loads.
  • The tuned model answers your test prompts better than the base.

Lab 29: Distributed training with Kubeflow Trainer

Goal: Run a training job across several GPUs, on one server and then across servers, and see the network cost.

Prerequisites

  • Lab 28
  • Training operator enabled

Environment notes

L40S cards are PCIe without NVLink. Without RDMA, traffic between servers goes over the pod network, so scaling across servers is limited.

Steps

  1. Run a TrainJob or PyTorchJob with one worker and one GPU.
  2. Run it with two GPUs on the same node.
  3. Run it with GPUs on two nodes.
  4. Record time per step for each.
  5. Explain the result in terms of the interconnect.

Verification

  • All three runs complete.
  • You have time-per-step numbers showing where scaling stops paying.

Lab 30: LoRA and QLoRA fine-tuning

Goal: Fine-tune an 8B model on a single L40S with parameter-efficient methods.

Prerequisites

  • Lab 28

Environment notes

Full fine-tuning of 8B needs more than 48 GB; LoRA usually fits, and QLoRA fits with room to spare.

Steps

  1. Prepare an instruction dataset.
  2. Run LoRA with the Training Hub or an InstructLab-style flow.
  3. Run QLoRA with 4-bit base weights.
  4. Compare memory, time and quality.
  5. Merge or serve the adapter.

Verification

  • Both runs fit on one GPU.
  • The adapted model improves on your evaluation set.

Lab 31: Ray and CodeFlare: Ray Train and Ray Tune

Goal: Use Ray for distributed training and hyperparameter search.

Prerequisites

  • Lab 28
  • Ray (KubeRay and CodeFlare) enabled

Environment notes

Size the Ray cluster with hardware profiles and queue it with Kueue.

Steps

  1. Create a RayCluster from a notebook with the CodeFlare SDK.
  2. Run a Ray Train job on two GPUs.
  3. Run a Ray Tune sweep over learning rate and batch size.
  4. Read the Ray dashboard.
  5. Tear the cluster down from the notebook.

Verification

  • The sweep finds a best trial.
  • Resources are released when the cluster is removed.

Lab 32: Kueue: quotas, fair share, preemption and borrowing

Goal: Share a small pool of GPUs between teams without arguments.

Prerequisites

  • Lab 5
  • Kueue enabled

Environment notes

With time-slicing on, quotas count shared GPU replicas, not physical cards.

Steps

  1. Create a ResourceFlavor for GPU nodes and a ClusterQueue with a GPU quota per team in a cohort.
  2. Create a LocalQueue in each team project.
  3. Submit more jobs than fit and watch them queue.
  4. Let one team borrow unused quota from another.
  5. Submit a high-priority job and watch preemption.

Verification

  • Jobs wait instead of failing when GPUs are busy.
  • Borrowing and preemption behave as configured.

Lab 33: Checkpointing to ONTAP and ODF

Goal: Save training state to persistent storage and resume after failure.

Prerequisites

  • Lab 28 or 29
  • RWX storage (Lab 9)

Environment notes

Checkpoints for large models are big; check volume size and write speed.

Steps

  1. Write checkpoints every N steps to a persistent volume.
  2. Delete the training pod mid-run.
  3. Resume from the latest checkpoint.
  4. Snapshot the checkpoint volume.
  5. Compare write times on NAS and Ceph storage.

Verification

  • The job resumes from the last checkpoint, not from step zero.
  • No more than N steps of work are lost.
← PreviousGen AI studio and RAGNext →Pipelines and MLOps