Combined Cloud Managed Services

User Tools

Site Tools

wiki:ai:nvidia_gpu_operator_installation_and_upgrade

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
wiki:ai:nvidia_gpu_operator_installation_and_upgrade [2026/07/20 10:58] – ssureshwiki:ai:nvidia_gpu_operator_installation_and_upgrade [2026/07/20 16:53] (current) – ssuresh
Line 180: Line 180:
 </code> </code>
  
 +===== Phase 4 — Validate =====
 +
 +<code ->
 +kubectl get nodes                       # must stay Ready
 +kubectl get pods -n gpu-operator
 +watch kubectl get pods -n gpu-operator  # wait for all Running/Completed; Ctrl+C to exit
 +</code>
 +
 +**Key check — GPU is schedulable:**
 +
 +<code ->
 +kubectl get nodes -o=jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.allocatable.nvidia\.com/gpu}{"\n"}{end}'
 +# Expect: <node>   1
 +</code>
 +
 +{{:wiki:ai:screenshot_2026-07-14_at_7.01.18 pm.png|}}
 +
 +
 +===== Phase 5 — CUDA VectorAdd GPU test =====
 +
 +<code ->
 +cat > cuda-vectoradd.yaml <<'EOF'
 +apiVersion: v1
 +kind: Pod
 +metadata:
 +  name: cuda-vectoradd
 +spec:
 +  restartPolicy: OnFailure
 +  containers:
 +  - name: cuda-vectoradd
 +    image: "nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda11.7.1-ubuntu20.04"
 +    resources:
 +      limits:
 +        nvidia.com/gpu: 1
 +EOF
 +
 +kubectl apply -f cuda-vectoradd.yaml
 +sleep 25
 +kubectl get pod cuda-vectoradd
 +kubectl logs pod/cuda-vectoradd     # success ends with: Test PASSED / Done
 +</code>
 +
 +{{:wiki:ai:screenshot_2026-07-14_at_7.05.33 pm.png|}}
 +
 +===== Phase 6 — Upgrade the Operator =====
 +
 +**Concept: **
 +
 +Helm upgrades everything except CRDs (it won't auto-upgrade existing CRDs). Real run performed **v26.3.1 → v26.3.3**.
 +
 +**6.1 Find your release name and available versions**
 +
 +<code ->
 +helm list -n gpu-operator                                # copy the real release name
 +helm search repo nvidia/gpu-operator --versions | head   # pick newest, e.g. v26.3.3
 +</code>
 +
 +**6.2 Apply updated CRDs**
 +
 +<code ->
 +export RELEASE_TAG=v26.3.3
 +kubectl apply -f https://raw.githubusercontent.com/NVIDIA/gpu-operator/refs/tags/$RELEASE_TAG/deployments/gpu-operator/crds/nvidia.com_clusterpolicies.yaml
 +kubectl apply -f https://raw.githubusercontent.com/NVIDIA/gpu-operator/refs/tags/$RELEASE_TAG/deployments/gpu-operator/crds/nvidia.com_nvidiadrivers.yaml
 +kubectl apply -f https://raw.githubusercontent.com/NVIDIA/gpu-operator/refs/tags/$RELEASE_TAG/deployments/gpu-operator/charts/node-feature-discovery/crds/nfd-api-crds.yaml
 +</code>
 +
 +("missing last-applied-configuration annotation" warnings are harmless.)
 +
 +**6.3 Upgrade (real release name + --reuse-values + re-passed k3s env)**
 +
 +<code ->
 +helm upgrade <YOUR_RELEASE_NAME> nvidia/gpu-operator -n gpu-operator \
 +    --version=$RELEASE_TAG \
 +    --reuse-values \
 +    --set toolkit.env[0].name=CONTAINERD_CONFIG \
 +    --set toolkit.env[0].value=/var/lib/rancher/k3s/agent/etc/containerd/config.toml \
 +    --set toolkit.env[1].name=CONTAINERD_SOCKET \
 +    --set toolkit.env[1].value=/run/k3s/containerd/containerd.sock \
 +    --set toolkit.env[2].name=CONTAINERD_RUNTIME_CLASS \
 +    --set toolkit.env[2].value=nvidia \
 +    --set-string toolkit.env[3].value=true \
 +    --set toolkit.env[3].name=CONTAINERD_SET_AS_DEFAULT
 +</code>
 +
 +**6.4 Watch the roll (expect a brief API blip)**
 +
 +<code ->
 +watch kubectl get pods -n gpu-operator
 +</code>
 +
 +{{:wiki:ai:screenshot_2026-07-14_at_7.09.52 pm.png|}}
 +
 +**6.5 Verify**
 +
 +<code ->
 +helm list -n gpu-operator     # new version, REVISION incremented
 +kubectl get nodes -o=jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.allocatable.nvidia\.com/gpu}{"\n"}{end}'   # still 1
 +kubectl apply -f cuda-vectoradd.yaml && sleep 25 && kubectl logs pod/cuda-vectoradd   # Test PASSED again
 +</code>
 +
 +{{:wiki:ai:screenshot_2026-07-14_at_7.17.10 pm.png|}}
 +
 +===== PART II — PRODUCTION UPGRADE SIMULATION =====
 +
 +Pretend the single EC2 node is a production AI cluster, run a live GPU workload, and walk through what an upgrade does to it — safely, without needing a real driver change.
 +
 +====Setup — Simulate GPU Operator Upgrade with Active ML Workloads====
 +
 +<code ->
 +NODE=$(kubectl get nodes -o jsonpath='{.items[0].metadata.name}')
 +echo $NODE
 +</code>
 +
 +==== A — Stand up a "production" GPU workload ====
 +
 +**A1. Deploy a real GPU workload (gpu-burn — genuine GPU load)**
 +
 +This runs gpu_burn, which actually stresses the GPU (near 100% utilization) — a far more realistic "training is running" stand-in than an idle pod.
 +
 +
 +<code ->
 +cat > gpu-burn.yaml <<'EOF'
 +apiVersion: v1
 +kind: Pod
 +metadata:
 +  name: gpu-burn
 +spec:
 +  restartPolicy: Never
 +  containers:
 +  - name: burn
 +    image: nvcr.io/nvidia/cuda:12.4.1-devel-ubuntu22.04
 +    command:
 +      - /bin/bash
 +      - -c
 +      - |
 +        apt-get update && \
 +        apt-get install -y git build-essential && \
 +        git clone https://github.com/wilicc/gpu-burn.git && \
 +        cd gpu-burn && \
 +        make && \
 +        ./gpu_burn 3600
 +    resources:
 +      limits:
 +        nvidia.com/gpu: 1
 +EOF
 +
 +kubectl apply -f gpu-burn.yaml
 +kubectl get pod gpu-burn -o wide      # wait for Running
 +</code>
 +
 +{{:wiki:ai:screenshot_2026-07-15_at_9.58.41 pm.png|}}
 +
 +**A2. Confirm the GPU is in use**
 +
 +Watch the build + burn progress, then check real GPU load from the driver pod:
 +
 +<code ->
 +kubectl logs gpu-burn -f      # apt install → clone → make → then GPU burn output; Ctrl+C to stop following
 +</code>
 +
 +{{:wiki:ai:screenshot_2026-07-15_at_9.58.54 pm.png|}}
 +
 +{{:wiki:ai:screenshot_2026-07-15_at_9.59.07 pm.png|}}
 +
 +<code ->
 +DRIVER=$(kubectl get pods -n gpu-operator -l app=nvidia-driver-daemonset -o jsonpath='{.items[0].metadata.name}')
 +kubectl exec -n gpu-operator $DRIVER -- nvidia-smi
 +</code>
 +
 +{{:wiki:ai:screenshot_2026-07-15_at_10.01.34 pm.png|}}
 +
 +Once burning, nvidia-smi shows GPU-Util climbing toward 100%, memory in use, and gpu_burn listed in the Processes section — genuine GPU activity, i.e. a real ML workload stand-in.
 +
 +==== B — Understand what an upgrade will do (before touching anything) ====
 +
 +**B1. Pre-upgrade check: what's using the GPU?**
 +
 +<code ->
 +kubectl get pods -A -o wide | grep -i gpu
 +kubectl get pods
 +</code>
 +
 +Always know what will be disrupted before upgrading.
 +
 +**B2. Inspect the Operator's built-in upgrade controls**
 +
 +<code ->
 +kubectl get clusterpolicies -o yaml | grep -A 25 "upgradePolicy"
 +kubectl get node $NODE -o jsonpath='{.metadata.labels.nvidia\.com/gpu-driver-upgrade-state}{"\n"}'
 +</code>
 +
 +**Pause / resume automatic driver upgrades cluster-wide:**
 +
 +<code ->
 +# Pause
 +kubectl patch clusterpolicies/cluster-policy --type merge \
 +  -p '{"spec":{"driver":{"upgradePolicy":{"autoUpgrade":false}}}}'
 +# Resume
 +kubectl patch clusterpolicies/cluster-policy --type merge \
 +  -p '{"spec":{"driver":{"upgradePolicy":{"autoUpgrade":true}}}}'
 +</code>
 +
 +{{:wiki:ai:screenshot_2026-07-15_at_10.04.48 pm.png|}}
 +
 +==== C — Simulate the disruptive part of a driver upgrade ====
 +
 +**C1. Cordon the node (maintenance mode)**
 +
 +<code ->
 +kubectl cordon $NODE
 +kubectl get node $NODE                  # STATUS: Ready,SchedulingDisabled
 +</code>
 +
 +{{:wiki:ai:screenshot_2026-07-16_at_11.05.13 pm.png|}}
 +{{:wiki:ai:screenshot_2026-07-16_at_11.05.54 pm.png|}}
 +
 +**C2. Drain the node → watch the workload get evicted**
 +
 +<code ->
 +kubectl drain $NODE --ignore-daemonsets --delete-emptydir-data --force
 +kubectl get pods                        # gpu-burn is gone (evicted)
 +</code>
 +
 +{{:wiki:ai:screenshot_2026-07-15_at_10.41.31 pm.png|}}
 +
 +==== D — Do the actual upgrade & validate ====
 +
 +**D1. Uncordon first so the upgrade schedules cleanly**
 +
 +<code ->
 +kubectl uncordon $NODE
 +kubectl get node $NODE                  # back to Ready
 +</code>
 +
 +**D2. Upgrade the GPU Operator**
 +
 +<code ->
 +helm list -n gpu-operator
 +helm search repo nvidia/gpu-operator --versions | head
 +</code>
 +
 +If a newer version exists, run the corrected upgrade (Part I, Phase 6.3). If already newest, you performed a real upgrade earlier in the story — the observation below is the point. Watch which pods restart:
 +
 +<code ->
 +kubectl get pods -n gpu-operator -w     # Ctrl+C when settled
 +</code>
 +
 +**D3. Validate after upgrade**
 +
 +<code ->
 +kubectl get pods -n gpu-operator
 +kubectl get nodes -o=jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.allocatable.nvidia\.com/gpu}{"\n"}{end}'   # expect 1
 +kubectl apply -f cuda-vectoradd.yaml
 +sleep 25
 +kubectl logs cuda-vectoradd             # expect: Test PASSED
 +kubectl delete -f cuda-vectoradd.yaml
 +</code>
 +
 +{{:wiki:ai:screenshot_2026-07-15_at_10.47.02 pm.png|}}
 +
 +==== E — Restore & learn the production lesson ====
 +
 +**E1. Restore the workload**
 +
 +<code ->
 +kubectl apply -f gpu-burn.yaml
 +kubectl get pods
 +</code>
 +
 +**E2. Inference with replicas + PodDisruptionBudget (why production has many nodes)**
 +
 +Free the GPU first so the inference pod can schedule:
 +
 +<code ->
 +kubectl delete -f gpu-burn.yaml --ignore-not-found
 +</code>
 +
 +Deploy an "inference" service with 2 replicas and a PDB that forbids dropping to zero:
 +
 +<code ->
 +cat > inference.yaml <<'EOF'
 +apiVersion: apps/v1
 +kind: Deployment
 +metadata:
 +  name: inference
 +spec:
 +  replicas: 2
 +  selector:
 +    matchLabels: { app: inference }
 +  template:
 +    metadata:
 +      labels: { app: inference }
 +    spec:
 +      containers:
 +      - name: inference
 +        image: nvcr.io/nvidia/cuda:12.4.1-base-ubuntu22.04
 +        command: ["bash","-c","sleep infinity"]
 +        resources:
 +          limits:
 +            nvidia.com/gpu: 1
 +---
 +apiVersion: policy/v1
 +kind: PodDisruptionBudget
 +metadata:
 +  name: inference-pdb
 +spec:
 +  minAvailable: 1
 +  selector:
 +    matchLabels: { app: inference }
 +EOF
 +
 +kubectl apply -f inference.yaml
 +kubectl get pods -l app=inference
 +</code>
 +
 +See the PDB protect the service — try to drain and watch it get blocked:
 +
 +<code ->
 +kubectl drain $NODE --ignore-daemonsets --delete-emptydir-data
 +# Blocked: "Cannot evict pod ... would violate the pod's disruption budget"
 +# Press Ctrl+C to stop the attempt
 +kubectl uncordon $NODE
 +</code>
 +
 +{{:wiki:ai:screenshot_2026-07-15_at_10.57.56 pm.png|}}
 +
 +
 +===== PART III — ROLLBACK PLAN =====
 +
 +**Before any upgrade — capture your safety net**
 +
 +<code ->
 +helm list -n gpu-operator                                             # note operator + version
 +helm get values <RELEASE_NAME> -n gpu-operator > working-values.yaml  # save WORKING config (incl. k3s toolkit.env)
 +helm history <RELEASE_NAME> -n gpu-operator                           # note current revision
 +</code>
 +
 +{{:wiki:ai:screenshot_2026-07-15_at_11.18.24 pm.png|}}
 +
 +
 +**Roll back the Operator release**
 +
 +<code ->
 +helm history <RELEASE_NAME> -n gpu-operator          # see revisions
 +helm rollback <RELEASE_NAME> -n gpu-operator          # previous revision
 +# or a specific revision:
 +helm rollback <RELEASE_NAME> 2 -n gpu-operator
 +helm rollback <RELEASE_NAME> 1 -n gpu-operator
 +</code>
 +
 +{{:wiki:ai:screenshot_2026-07-15_at_11.19.40 pm.png|}}
 +
 +**Simulation cleanup**
 +
 +<code ->
 +kubectl delete -f gpu-burn.yaml --ignore-not-found
 +kubectl delete -f inference.yaml --ignore-not-found
 +kubectl delete pdb inference-pdb --ignore-not-found
 +kubectl uncordon $NODE 2>/dev/null
 +kubectl get pods
 +</code>
 +
 +{{:wiki:ai:screenshot_2026-07-15_at_10.59.54 pm.png|}}
 +
 +**Cleanup (avoid a surprise bill!)**
 +
 +  * Pause between sessions: EC2 → Stop (compute stops billing; small EBS cost remains; public IP changes on restart unless you use an Elastic IP).
 +  * Done for good: EC2 → Terminate (deletes instance + disk).
 +  * Release any Elastic IP if you allocated one (billed even while unused).
wiki/ai/nvidia_gpu_operator_installation_and_upgrade.1784545106.txt.gz · Last modified: by ssuresh

Page Tools

Trace:
Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com) IP:10.0.158.18