====== NVIDIA GPU Operator Installation and Upgrade ======

===== Overview =====

This document captures a complete hands-on lab for the NVIDIA GPU Operator, covering the full lifecycle: installation, validation, GPU workload testing, upgrade, and rollback. The environment is a self-built single-node Kubernetes cluster (k3s) running on an AWS [g4dn.xlarge] instance with an NVIDIA T4 GPU. The goal was to understand how GPUs are enabled and managed in Kubernetes .Kubernetes has no native GPU awareness, and the GPU Operator automates the driver, container toolkit, and device plugin required to make a GPU schedulable. Beyond the core lab, the work was extended into a production upgrade simulation (upgrading with live ML workloads, node drain/eviction behaviour, PodDisruptionBudgets, and multi-node reasoning) and a documented rollback plan. Real issues encountered most notably a k3s CNI failure caused by the container config path are documented with root cause and fix.

===== PART I — LAB RUNBOOK =====

===== Phase 1 — Launch the EC2 instance =====

Console → **EC2** → **Launch instance**:

| **Setting** | **Value** |
| **Name** | gpu-operator-lab |
| **AMI** | Ubuntu Server 24.04 LTS (HVM), SSD Volume Type (standard — NOT Deep Learning) |
| **Instance type** | g4dn.xlarge |
| **Key pair** | your key pair |
| **Storage** | **100 GB** gp3 (GPU container images are large) |
| **Security group** | Allow **SSH (22)** from //My IP// only |

Launch, then note the instance's public IP.

{{:wiki:ai:screenshot_2026-07-14_at_3.48.08 pm.png}}

{{:wiki:ai:screenshot_2026-07-13_at_9.38.39 pm.png}}

{{:wiki:ai:screenshot_2026-07-13_at_10.05.44 pm.png}}

**__Note:__**

**Before creating or updating the EC2 Security Group Rule:**

  - Turn **OFF Tailscale**.
  - Turn **OFF Zscaler**.
  - Go to **EC2 → Instance → Security → Security Group → Edit inbound rules**.
  - Set **SSH (TCP 22)** source to **My IP**.
  - Save the rule.
  - If you accidentally used the wrong IP, disable Tailscale and Zscaler, then update the security group with **My IP** again.

===== Phase 2 — Connect, verify GPU, install k3s =====

**2.1 SSH in**

<code ->
chmod 400 /path/to/your-key.pem
ssh -i /path/to/your-key.pem ubuntu@<PUBLIC_IP>
</code>

{{:wiki:ai:screenshot_2026-07-14_at_4.36.45 pm.png|}}

Accept the host-key prompt with yes.

**2.2 Confirm the GPU hardware is present (do NOT install a driver manually)**

<code ->
lspci | grep -i nvidia
</code>

{{:wiki:ai:screenshot_2026-07-14_at_6.50.18 pm.png|}}

You should see an NVIDIA Tesla T4 line. If nothing appears, stop the instance has no GPU and nothing else will work.

**2.3 Install jq**

<code ->
sudo apt-get update && sudo apt-get install -y jq
</code>

{{:wiki:ai:screenshot_2026-07-14_at_6.50.34 pm.png|}}

**2.4 Install k3s**

<code ->
curl -sfL https://get.k3s.io | sh -
</code>

**2.5 Make kubectl usable without sudo**

<code ->
mkdir -p ~/.kube
sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config
sudo chown $(id -u):$(id -g) ~/.kube/config
export KUBECONFIG=~/.kube/config
echo 'export KUBECONFIG=~/.kube/config' >> ~/.bashrc
</code>

{{:wiki:ai:screenshot_2026-07-14_at_6.51.40 pm.png|}}

**2.6 Confirm the cluster is up**

<code ->
kubectl get nodes
</code>

**2.7 Install Helm**

<code ->
curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3

chmod 700 get_helm.sh && ./get_helm.sh

helm version
</code>

{{:wiki:ai:screenshot_2026-07-14_at_6.53.29 pm.png|}}

===== Phase 3 — Install the GPU Operator =====

**3.1 Add the NVIDIA Helm repo**

<code ->
helm repo add nvidia https://helm.ngc.nvidia.com/nvidia && helm repo update
</code>


**3.2 Install (uses config.toml, not config.toml.tmpl)**

<code ->
helm install --wait --generate-name \
    -n gpu-operator --create-namespace \
    nvidia/gpu-operator \
    --version=v26.3.1 \
    --set toolkit.env[0].name=CONTAINERD_CONFIG \
    --set toolkit.env[0].value=/var/lib/rancher/k3s/agent/etc/containerd/config.toml \
    --set toolkit.env[1].name=CONTAINERD_SOCKET \
    --set toolkit.env[1].value=/run/k3s/containerd/containerd.sock \
    --set toolkit.env[2].name=CONTAINERD_RUNTIME_CLASS \
    --set toolkit.env[2].value=nvidia \
    --set-string toolkit.env[3].value=true \
    --set toolkit.env[3].name=CONTAINERD_SET_AS_DEFAULT
</code>

{{:wiki:ai:screenshot_2026-07-14_at_6.58.29 pm.png|}}

**The critical fix:** CONTAINERD_CONFIG must be config.toml. Using config.toml.tmpl overwrites k3s's container template and drops its CNI (flannel) config → node NotReady, pods Unknown/Init/Pending. 

{{:wiki:ai:screenshot_2026-07-14_at_1.03.44 am.png|}}

===== DEEP-DIVE: THE k3s CNI ISSUE EXPLAINED =====

==== The setup ====

k3s (the lightweight Kubernetes) runs a component called containerd — the thing that actually starts and stops containers. containerd reads its settings from a file called config.toml.
k3s generates that config.toml automatically from a template file called config.toml.tmpl. Think of it like:
config.toml.tmpl (the template/recipe) → k3s uses it to produce → config.toml (the actual settings containerd reads)
Crucially, k3s's template already contains the CNI (networking) settings — CNI is what gives pods their network and lets the node be "ready." Without CNI, the node can't function.

==== What went wrong ====

When you installed the GPU Operator, its toolkit needs to add the "nvidia runtime" into containerd's config. The original command told the toolkit to write into config.toml.tmpl (the template).
The toolkit overwrote that template with its own version — and its version did not include k3s's CNI settings. So the next time k3s regenerated config.toml from the now-broken template, the networking config was gone.

**Result, step by step:**

•	CNI settings lost → "cni plugin not initialized"

•	No networking → node goes NotReady

•	A NotReady node can't place pods → everything stuck Unknown/Pending

•	Nothing scheduled → the GPU never became usable


That's the "broke the node → NotReady → nothing scheduled" chain.

==== The fix ====

Point the toolkit at config.toml (the actual generated file) instead of config.toml.tmpl (the template).

**Why that works:** writing to config.toml makes the toolkit add the nvidia runtime to the file that already has k3s's CNI settings — so CNI is preserved and the nvidia runtime is added alongside it. Nothing gets wiped.

==== Recovery steps (if you already broke it) ====

<code ->
helm uninstall <RELEASE_NAME> -n gpu-operator
sudo rm -f /var/lib/rancher/k3s/agent/etc/containerd/config.toml.tmpl
sudo systemctl restart k3s
kubectl get nodes        # wait until Ready again
# then reinstall using config.toml (Part II, Phase 3.2)
</code>

===== Phase 4 — Validate =====

<code ->
kubectl get nodes                       # must stay Ready
kubectl get pods -n gpu-operator
watch kubectl get pods -n gpu-operator  # wait for all Running/Completed; Ctrl+C to exit
</code>

**Key check — GPU is schedulable:**

<code ->
kubectl get nodes -o=jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.allocatable.nvidia\.com/gpu}{"\n"}{end}'
# Expect: <node>   1
</code>

{{:wiki:ai:screenshot_2026-07-14_at_7.01.18 pm.png|}}


===== Phase 5 — CUDA VectorAdd GPU test =====

<code ->
cat > cuda-vectoradd.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: cuda-vectoradd
spec:
  restartPolicy: OnFailure
  containers:
  - name: cuda-vectoradd
    image: "nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda11.7.1-ubuntu20.04"
    resources:
      limits:
        nvidia.com/gpu: 1
EOF

kubectl apply -f cuda-vectoradd.yaml
sleep 25
kubectl get pod cuda-vectoradd
kubectl logs pod/cuda-vectoradd     # success ends with: Test PASSED / Done
</code>

{{:wiki:ai:screenshot_2026-07-14_at_7.05.33 pm.png|}}

===== Phase 6 — Upgrade the Operator =====

**Concept: **

Helm upgrades everything except CRDs (it won't auto-upgrade existing CRDs). Real run performed **v26.3.1 → v26.3.3**.

**6.1 Find your release name and available versions**

<code ->
helm list -n gpu-operator                                # copy the real release name
helm search repo nvidia/gpu-operator --versions | head   # pick newest, e.g. v26.3.3
</code>

**6.2 Apply updated CRDs**

<code ->
export RELEASE_TAG=v26.3.3
kubectl apply -f https://raw.githubusercontent.com/NVIDIA/gpu-operator/refs/tags/$RELEASE_TAG/deployments/gpu-operator/crds/nvidia.com_clusterpolicies.yaml
kubectl apply -f https://raw.githubusercontent.com/NVIDIA/gpu-operator/refs/tags/$RELEASE_TAG/deployments/gpu-operator/crds/nvidia.com_nvidiadrivers.yaml
kubectl apply -f https://raw.githubusercontent.com/NVIDIA/gpu-operator/refs/tags/$RELEASE_TAG/deployments/gpu-operator/charts/node-feature-discovery/crds/nfd-api-crds.yaml
</code>

("missing last-applied-configuration annotation" warnings are harmless.)

**6.3 Upgrade (real release name + --reuse-values + re-passed k3s env)**

<code ->
helm upgrade <YOUR_RELEASE_NAME> nvidia/gpu-operator -n gpu-operator \
    --version=$RELEASE_TAG \
    --reuse-values \
    --set toolkit.env[0].name=CONTAINERD_CONFIG \
    --set toolkit.env[0].value=/var/lib/rancher/k3s/agent/etc/containerd/config.toml \
    --set toolkit.env[1].name=CONTAINERD_SOCKET \
    --set toolkit.env[1].value=/run/k3s/containerd/containerd.sock \
    --set toolkit.env[2].name=CONTAINERD_RUNTIME_CLASS \
    --set toolkit.env[2].value=nvidia \
    --set-string toolkit.env[3].value=true \
    --set toolkit.env[3].name=CONTAINERD_SET_AS_DEFAULT
</code>

**6.4 Watch the roll (expect a brief API blip)**

<code ->
watch kubectl get pods -n gpu-operator
</code>

{{:wiki:ai:screenshot_2026-07-14_at_7.09.52 pm.png|}}

**6.5 Verify**

<code ->
helm list -n gpu-operator     # new version, REVISION incremented
kubectl get nodes -o=jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.allocatable.nvidia\.com/gpu}{"\n"}{end}'   # still 1
kubectl apply -f cuda-vectoradd.yaml && sleep 25 && kubectl logs pod/cuda-vectoradd   # Test PASSED again
</code>

{{:wiki:ai:screenshot_2026-07-14_at_7.17.10 pm.png|}}

===== PART II — PRODUCTION UPGRADE SIMULATION =====

Pretend the single EC2 node is a production AI cluster, run a live GPU workload, and walk through what an upgrade does to it — safely, without needing a real driver change.

====Setup — Simulate GPU Operator Upgrade with Active ML Workloads====

<code ->
NODE=$(kubectl get nodes -o jsonpath='{.items[0].metadata.name}')
echo $NODE
</code>

==== A — Stand up a "production" GPU workload ====

**A1. Deploy a real GPU workload (gpu-burn — genuine GPU load)**

This runs gpu_burn, which actually stresses the GPU (near 100% utilization) — a far more realistic "training is running" stand-in than an idle pod.


<code ->
cat > gpu-burn.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: gpu-burn
spec:
  restartPolicy: Never
  containers:
  - name: burn
    image: nvcr.io/nvidia/cuda:12.4.1-devel-ubuntu22.04
    command:
      - /bin/bash
      - -c
      - |
        apt-get update && \
        apt-get install -y git build-essential && \
        git clone https://github.com/wilicc/gpu-burn.git && \
        cd gpu-burn && \
        make && \
        ./gpu_burn 3600
    resources:
      limits:
        nvidia.com/gpu: 1
EOF

kubectl apply -f gpu-burn.yaml
kubectl get pod gpu-burn -o wide      # wait for Running
</code>

{{:wiki:ai:screenshot_2026-07-15_at_9.58.41 pm.png|}}

**A2. Confirm the GPU is in use**

Watch the build + burn progress, then check real GPU load from the driver pod:

<code ->
kubectl logs gpu-burn -f      # apt install → clone → make → then GPU burn output; Ctrl+C to stop following
</code>

{{:wiki:ai:screenshot_2026-07-15_at_9.58.54 pm.png|}}

{{:wiki:ai:screenshot_2026-07-15_at_9.59.07 pm.png|}}

<code ->
DRIVER=$(kubectl get pods -n gpu-operator -l app=nvidia-driver-daemonset -o jsonpath='{.items[0].metadata.name}')
kubectl exec -n gpu-operator $DRIVER -- nvidia-smi
</code>

{{:wiki:ai:screenshot_2026-07-15_at_10.01.34 pm.png|}}

Once burning, nvidia-smi shows GPU-Util climbing toward 100%, memory in use, and gpu_burn listed in the Processes section — genuine GPU activity, i.e. a real ML workload stand-in.

==== B — Understand what an upgrade will do (before touching anything) ====

**B1. Pre-upgrade check: what's using the GPU?**

<code ->
kubectl get pods -A -o wide | grep -i gpu
kubectl get pods
</code>

Always know what will be disrupted before upgrading.

**B2. Inspect the Operator's built-in upgrade controls**

<code ->
kubectl get clusterpolicies -o yaml | grep -A 25 "upgradePolicy"
kubectl get node $NODE -o jsonpath='{.metadata.labels.nvidia\.com/gpu-driver-upgrade-state}{"\n"}'
</code>

**Pause / resume automatic driver upgrades cluster-wide:**

<code ->
# Pause
kubectl patch clusterpolicies/cluster-policy --type merge \
  -p '{"spec":{"driver":{"upgradePolicy":{"autoUpgrade":false}}}}'
# Resume
kubectl patch clusterpolicies/cluster-policy --type merge \
  -p '{"spec":{"driver":{"upgradePolicy":{"autoUpgrade":true}}}}'
</code>

{{:wiki:ai:screenshot_2026-07-15_at_10.04.48 pm.png|}}

==== C — Simulate the disruptive part of a driver upgrade ====

**C1. Cordon the node (maintenance mode)**

<code ->
kubectl cordon $NODE
kubectl get node $NODE                  # STATUS: Ready,SchedulingDisabled
</code>

{{:wiki:ai:screenshot_2026-07-16_at_11.05.13 pm.png|}}
{{:wiki:ai:screenshot_2026-07-16_at_11.05.54 pm.png|}}

**C2. Drain the node → watch the workload get evicted**

<code ->
kubectl drain $NODE --ignore-daemonsets --delete-emptydir-data --force
kubectl get pods                        # gpu-burn is gone (evicted)
</code>

{{:wiki:ai:screenshot_2026-07-15_at_10.41.31 pm.png|}}

==== D — Do the actual upgrade & validate ====

**D1. Uncordon first so the upgrade schedules cleanly**

<code ->
kubectl uncordon $NODE
kubectl get node $NODE                  # back to Ready
</code>

**D2. Upgrade the GPU Operator**

<code ->
helm list -n gpu-operator
helm search repo nvidia/gpu-operator --versions | head
</code>

If a newer version exists, run the corrected upgrade (Part I, Phase 6.3). If already newest, you performed a real upgrade earlier in the story — the observation below is the point. Watch which pods restart:

<code ->
kubectl get pods -n gpu-operator -w     # Ctrl+C when settled
</code>

**D3. Validate after upgrade**

<code ->
kubectl get pods -n gpu-operator
kubectl get nodes -o=jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.allocatable.nvidia\.com/gpu}{"\n"}{end}'   # expect 1
kubectl apply -f cuda-vectoradd.yaml
sleep 25
kubectl logs cuda-vectoradd             # expect: Test PASSED
kubectl delete -f cuda-vectoradd.yaml
</code>

{{:wiki:ai:screenshot_2026-07-15_at_10.47.02 pm.png|}}

==== E — Restore & learn the production lesson ====

**E1. Restore the workload**

<code ->
kubectl apply -f gpu-burn.yaml
kubectl get pods
</code>

**E2. Inference with replicas + PodDisruptionBudget (why production has many nodes)**

Free the GPU first so the inference pod can schedule:

<code ->
kubectl delete -f gpu-burn.yaml --ignore-not-found
</code>

Deploy an "inference" service with 2 replicas and a PDB that forbids dropping to zero:

<code ->
cat > inference.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: inference
spec:
  replicas: 2
  selector:
    matchLabels: { app: inference }
  template:
    metadata:
      labels: { app: inference }
    spec:
      containers:
      - name: inference
        image: nvcr.io/nvidia/cuda:12.4.1-base-ubuntu22.04
        command: ["bash","-c","sleep infinity"]
        resources:
          limits:
            nvidia.com/gpu: 1
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: inference-pdb
spec:
  minAvailable: 1
  selector:
    matchLabels: { app: inference }
EOF

kubectl apply -f inference.yaml
kubectl get pods -l app=inference
</code>

See the PDB protect the service — try to drain and watch it get blocked:

<code ->
kubectl drain $NODE --ignore-daemonsets --delete-emptydir-data
# Blocked: "Cannot evict pod ... would violate the pod's disruption budget"
# Press Ctrl+C to stop the attempt
kubectl uncordon $NODE
</code>

{{:wiki:ai:screenshot_2026-07-15_at_10.57.56 pm.png|}}


===== PART III — ROLLBACK PLAN =====

**Before any upgrade — capture your safety net**

<code ->
helm list -n gpu-operator                                             # note operator + version
helm get values <RELEASE_NAME> -n gpu-operator > working-values.yaml  # save WORKING config (incl. k3s toolkit.env)
helm history <RELEASE_NAME> -n gpu-operator                           # note current revision
</code>

{{:wiki:ai:screenshot_2026-07-15_at_11.18.24 pm.png|}}


**Roll back the Operator release**

<code ->
helm history <RELEASE_NAME> -n gpu-operator          # see revisions
helm rollback <RELEASE_NAME> -n gpu-operator          # previous revision
# or a specific revision:
helm rollback <RELEASE_NAME> 2 -n gpu-operator
helm rollback <RELEASE_NAME> 1 -n gpu-operator
</code>

{{:wiki:ai:screenshot_2026-07-15_at_11.19.40 pm.png|}}

**Simulation cleanup**

<code ->
kubectl delete -f gpu-burn.yaml --ignore-not-found
kubectl delete -f inference.yaml --ignore-not-found
kubectl delete pdb inference-pdb --ignore-not-found
kubectl uncordon $NODE 2>/dev/null
kubectl get pods
</code>

{{:wiki:ai:screenshot_2026-07-15_at_10.59.54 pm.png|}}

**Cleanup (avoid a surprise bill!)**

  * Pause between sessions: EC2 → Stop (compute stops billing; small EBS cost remains; public IP changes on restart unless you use an Elastic IP).
  * Done for good: EC2 → Terminate (deletes instance + disk).
  * Release any Elastic IP if you allocated one (billed even while unused).