DC/DR and resilience

With two sites, AI services can survive the loss of one. These labs build and test that, and measure how long recovery really takes.

Every lab assumes the environment described in the series overview. Commands and names are examples: replace them with your own.

Labs in this section

• Lab 51: Model failover and RTO
• Lab 52: Replicating model storage with SnapMirror
• Lab 53: Platform failover: identity, registry and MLflow
• Lab 54: Backup and restore with OADP
• Lab 55: Multi-cluster mesh and Submariner

Lab 51: Model failover and RTO

Goal: Serve one model at both sites and measure recovery time when the primary fails.

Prerequisites

• Labs 12 and 40
• A DNS or global load balancer you control

Environment notes

Health checks should test a real inference, not just a TCP port.

Steps

1. Deploy the same model at both sites from GitOps.
2. Put a DNS name or load balancer in front.
3. Stop the model at the primary site.
4. Switch traffic and measure time until answers resume.
5. Switch back.

Verification

• Clients recover within your RTO target.
• The measured RTO is recorded.

Lab 52: Replicating model storage with SnapMirror

Goal: Replicate model weights and data to the second site with ONTAP SnapMirror.

Prerequisites

• NetApp ONTAP at both sites with Trident
• Lab 9

Environment notes

Replicated volumes are read-only at the destination until broken off or promoted.

Steps

1. Set up a SnapMirror relationship for the model volume.
2. Write a new model version at the source and wait for replication.
3. Break the relationship at the destination and mount the volume.
4. Serve the model from it.
5. Resync after the test.

Verification

• The destination serves the replicated model.
• The resync completes without a full copy.

Lab 53: Platform failover: identity, registry and MLflow

Goal: Keep sign-in, the model registry and experiment tracking working when a site fails.

Prerequisites

• Labs 36–37
• An identity provider deployed at both sites

Environment notes

Databases behind the registry and MLflow need replication or restore; decide which is acceptable.

Steps

1. List every stateful dependency: IdP, registry database, MLflow database, pipeline buckets.
2. Set up replication or scheduled backup for each.
3. Fail over the IdP and sign in at the second site.
4. Restore or promote the databases and check data.
5. Write a runbook with order and owners.

Verification

• Users sign in and see their models at the second site.
• The runbook is tested end to end.

Lab 54: Backup and restore with OADP

Goal: Back up OpenShift AI projects at one site and restore them at the other.

Prerequisites

• OADP installed at both sites with a shared S3 location

Environment notes

Back up namespaces and volumes; operator-managed resources are recreated by the operator.

Steps

1. Create a Backup of a data science project including volumes.
2. Check the backup in object storage.
3. Restore it at the other site.
4. Start the workbench and model server there.
5. Schedule regular backups.

Verification

• The restored project works at the second site.
• Scheduled backups complete.

Lab 55: Multi-cluster mesh and Submariner

Goal: Connect services across sites for future active-active designs.

Prerequisites

• Service Mesh 3 at both sites with a shared root CA
• Submariner (optional)

Environment notes

Plan the MTU for any tunnel between sites, and keep latency-sensitive traffic local.

Steps

1. Confirm both meshes trust the same root CA.
2. Set up east-west gateways and remote secrets.
3. Call a service at the other site through the mesh.
4. Install Submariner and export a service.
5. Measure latency across sites.

Verification

• Cross-site calls succeed with mTLS.
• Latency is measured and documented.