This document captures a complete hands-on lab for the NVIDIA GPU Operator, covering the full lifecycle: installation, validation, GPU workload testing, upgrade, and rollback. The environment is a self-built single-node Kubernetes cluster (k3s) running on an AWS [g4dn.xlarge] instance with an NVIDIA T4 GPU. The goal was to understand how GPUs are enabled and managed in Kubernetes .Kubernetes has no native GPU awareness, and the GPU Operator automates the driver, container toolkit, and device plugin required to make a GPU schedulable. Beyond the core lab, the work was extended into a production upgrade simulation (upgrading with live ML workloads, node drain/eviction behaviour, PodDisruptionBudgets, and multi-node reasoning) and a documented rollback plan. Real issues encountered most notably a k3s CNI failure caused by the container config path are documented with root cause and fix.

Console → EC2 → Launch instance:

Launch, then note the instance's public IP.

Note:

Before creating or updating the EC2 Security Group Rule:

1. Turn OFF Tailscale.
2. Turn OFF Zscaler.
3. Go to EC2 → Instance → Security → Security Group → Edit inbound rules.
4. Set SSH (TCP 22) source to My IP.
5. Save the rule.
6. If you accidentally used the wrong IP, disable Tailscale and Zscaler, then update the security group with My IP again.

2.1 SSH in

chmod 400 /path/to/your-key.pem
ssh -i /path/to/your-key.pem ubuntu@<PUBLIC_IP>

Accept the host-key prompt with yes.

2.2 Confirm the GPU hardware is present (do NOT install a driver manually)

lspci | grep -i nvidia

You should see an NVIDIA Tesla T4 line. If nothing appears, stop the instance has no GPU and nothing else will work.

2.3 Install jq

sudo apt-get update && sudo apt-get install -y jq

2.4 Install k3s

curl -sfL https://get.k3s.io | sh -

2.5 Make kubectl usable without sudo

mkdir -p ~/.kube
sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config
sudo chown $(id -u):$(id -g) ~/.kube/config
export KUBECONFIG=~/.kube/config
echo 'export KUBECONFIG=~/.kube/config' >> ~/.bashrc

2.6 Confirm the cluster is up

kubectl get nodes

2.7 Install Helm

curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3

chmod 700 get_helm.sh && ./get_helm.sh

helm version

3.1 Add the NVIDIA Helm repo

helm repo add nvidia https://helm.ngc.nvidia.com/nvidia && helm repo update

3.2 Install (uses config.toml, not config.toml.tmpl)

helm install --wait --generate-name \
    -n gpu-operator --create-namespace \
    nvidia/gpu-operator \
    --version=v26.3.1 \
    --set toolkit.env[0].name=CONTAINERD_CONFIG \
    --set toolkit.env[0].value=/var/lib/rancher/k3s/agent/etc/containerd/config.toml \
    --set toolkit.env[1].name=CONTAINERD_SOCKET \
    --set toolkit.env[1].value=/run/k3s/containerd/containerd.sock \
    --set toolkit.env[2].name=CONTAINERD_RUNTIME_CLASS \
    --set toolkit.env[2].value=nvidia \
    --set-string toolkit.env[3].value=true \
    --set toolkit.env[3].name=CONTAINERD_SET_AS_DEFAULT

The critical fix: CONTAINERD_CONFIG must be config.toml. Using config.toml.tmpl overwrites k3s's container template and drops its CNI (flannel) config → node NotReady, pods Unknown/Init/Pending.

k3s (the lightweight Kubernetes) runs a component called containerd — the thing that actually starts and stops containers. containerd reads its settings from a file called config.toml. k3s generates that config.toml automatically from a template file called config.toml.tmpl. Think of it like: config.toml.tmpl (the template/recipe) → k3s uses it to produce → config.toml (the actual settings containerd reads) Crucially, k3s's template already contains the CNI (networking) settings — CNI is what gives pods their network and lets the node be “ready.” Without CNI, the node can't function.

When you installed the GPU Operator, its toolkit needs to add the “nvidia runtime” into containerd's config. The original command told the toolkit to write into config.toml.tmpl (the template). The toolkit overwrote that template with its own version — and its version did not include k3s's CNI settings. So the next time k3s regenerated config.toml from the now-broken template, the networking config was gone.

Result, step by step:

• CNI settings lost → “cni plugin not initialized”

• No networking → node goes NotReady

• A NotReady node can't place pods → everything stuck Unknown/Pending

• Nothing scheduled → the GPU never became usable

That's the “broke the node → NotReady → nothing scheduled” chain.

Point the toolkit at config.toml (the actual generated file) instead of config.toml.tmpl (the template).

Why that works: writing to config.toml makes the toolkit add the nvidia runtime to the file that already has k3s's CNI settings — so CNI is preserved and the nvidia runtime is added alongside it. Nothing gets wiped.

helm uninstall <RELEASE_NAME> -n gpu-operator
sudo rm -f /var/lib/rancher/k3s/agent/etc/containerd/config.toml.tmpl
sudo systemctl restart k3s
kubectl get nodes        # wait until Ready again
# then reinstall using config.toml (Part II, Phase 3.2)

kubectl get nodes                       # must stay Ready
kubectl get pods -n gpu-operator
watch kubectl get pods -n gpu-operator  # wait for all Running/Completed; Ctrl+C to exit

Key check — GPU is schedulable:

kubectl get nodes -o=jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.allocatable.nvidia\.com/gpu}{"\n"}{end}'
# Expect: <node>   1

cat > cuda-vectoradd.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: cuda-vectoradd
spec:
  restartPolicy: OnFailure
  containers:
  - name: cuda-vectoradd
    image: "nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda11.7.1-ubuntu20.04"
    resources:
      limits:
        nvidia.com/gpu: 1
EOF

kubectl apply -f cuda-vectoradd.yaml
sleep 25
kubectl get pod cuda-vectoradd
kubectl logs pod/cuda-vectoradd     # success ends with: Test PASSED / Done

**Concept: **

Helm upgrades everything except CRDs (it won't auto-upgrade existing CRDs). Real run performed v26.3.1 → v26.3.3.

6.1 Find your release name and available versions

helm list -n gpu-operator                                # copy the real release name
helm search repo nvidia/gpu-operator --versions | head   # pick newest, e.g. v26.3.3

6.2 Apply updated CRDs

export RELEASE_TAG=v26.3.3
kubectl apply -f https://raw.githubusercontent.com/NVIDIA/gpu-operator/refs/tags/$RELEASE_TAG/deployments/gpu-operator/crds/nvidia.com_clusterpolicies.yaml
kubectl apply -f https://raw.githubusercontent.com/NVIDIA/gpu-operator/refs/tags/$RELEASE_TAG/deployments/gpu-operator/crds/nvidia.com_nvidiadrivers.yaml
kubectl apply -f https://raw.githubusercontent.com/NVIDIA/gpu-operator/refs/tags/$RELEASE_TAG/deployments/gpu-operator/charts/node-feature-discovery/crds/nfd-api-crds.yaml

(“missing last-applied-configuration annotation” warnings are harmless.)

6.3 Upgrade (real release name + –reuse-values + re-passed k3s env)

helm upgrade <YOUR_RELEASE_NAME> nvidia/gpu-operator -n gpu-operator \
    --version=$RELEASE_TAG \
    --reuse-values \
    --set toolkit.env[0].name=CONTAINERD_CONFIG \
    --set toolkit.env[0].value=/var/lib/rancher/k3s/agent/etc/containerd/config.toml \
    --set toolkit.env[1].name=CONTAINERD_SOCKET \
    --set toolkit.env[1].value=/run/k3s/containerd/containerd.sock \
    --set toolkit.env[2].name=CONTAINERD_RUNTIME_CLASS \
    --set toolkit.env[2].value=nvidia \
    --set-string toolkit.env[3].value=true \
    --set toolkit.env[3].name=CONTAINERD_SET_AS_DEFAULT

6.4 Watch the roll (expect a brief API blip)

watch kubectl get pods -n gpu-operator

6.5 Verify

helm list -n gpu-operator     # new version, REVISION incremented
kubectl get nodes -o=jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.allocatable.nvidia\.com/gpu}{"\n"}{end}'   # still 1
kubectl apply -f cuda-vectoradd.yaml && sleep 25 && kubectl logs pod/cuda-vectoradd   # Test PASSED again

Pretend the single EC2 node is a production AI cluster, run a live GPU workload, and walk through what an upgrade does to it — safely, without needing a real driver change.

NODE=$(kubectl get nodes -o jsonpath='{.items[0].metadata.name}')
echo $NODE

A1. Deploy a real GPU workload (gpu-burn — genuine GPU load)

This runs gpu_burn, which actually stresses the GPU (near 100% utilization) — a far more realistic “training is running” stand-in than an idle pod.

cat > gpu-burn.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: gpu-burn
spec:
  restartPolicy: Never
  containers:
  - name: burn
    image: nvcr.io/nvidia/cuda:12.4.1-devel-ubuntu22.04
    command:
      - /bin/bash
      - -c
      - |
        apt-get update && \
        apt-get install -y git build-essential && \
        git clone https://github.com/wilicc/gpu-burn.git && \
        cd gpu-burn && \
        make && \
        ./gpu_burn 3600
    resources:
      limits:
        nvidia.com/gpu: 1
EOF

kubectl apply -f gpu-burn.yaml
kubectl get pod gpu-burn -o wide      # wait for Running

A2. Confirm the GPU is in use

Watch the build + burn progress, then check real GPU load from the driver pod:

kubectl logs gpu-burn -f      # apt install → clone → make → then GPU burn output; Ctrl+C to stop following

DRIVER=$(kubectl get pods -n gpu-operator -l app=nvidia-driver-daemonset -o jsonpath='{.items[0].metadata.name}')
kubectl exec -n gpu-operator $DRIVER -- nvidia-smi

Once burning, nvidia-smi shows GPU-Util climbing toward 100%, memory in use, and gpu_burn listed in the Processes section — genuine GPU activity, i.e. a real ML workload stand-in.

B1. Pre-upgrade check: what's using the GPU?

kubectl get pods -A -o wide | grep -i gpu
kubectl get pods

Always know what will be disrupted before upgrading.

B2. Inspect the Operator's built-in upgrade controls

kubectl get clusterpolicies -o yaml | grep -A 25 "upgradePolicy"
kubectl get node $NODE -o jsonpath='{.metadata.labels.nvidia\.com/gpu-driver-upgrade-state}{"\n"}'

Pause / resume automatic driver upgrades cluster-wide:

# Pause
kubectl patch clusterpolicies/cluster-policy --type merge \
  -p '{"spec":{"driver":{"upgradePolicy":{"autoUpgrade":false}}}}'
# Resume
kubectl patch clusterpolicies/cluster-policy --type merge \
  -p '{"spec":{"driver":{"upgradePolicy":{"autoUpgrade":true}}}}'

C1. Cordon the node (maintenance mode)

kubectl cordon $NODE
kubectl get node $NODE                  # STATUS: Ready,SchedulingDisabled

C2. Drain the node → watch the workload get evicted

kubectl drain $NODE --ignore-daemonsets --delete-emptydir-data --force
kubectl get pods                        # gpu-burn is gone (evicted)

D1. Uncordon first so the upgrade schedules cleanly

kubectl uncordon $NODE
kubectl get node $NODE                  # back to Ready

D2. Upgrade the GPU Operator

helm list -n gpu-operator
helm search repo nvidia/gpu-operator --versions | head

If a newer version exists, run the corrected upgrade (Part I, Phase 6.3). If already newest, you performed a real upgrade earlier in the story — the observation below is the point. Watch which pods restart:

kubectl get pods -n gpu-operator -w     # Ctrl+C when settled

D3. Validate after upgrade

kubectl get pods -n gpu-operator
kubectl get nodes -o=jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.allocatable.nvidia\.com/gpu}{"\n"}{end}'   # expect 1
kubectl apply -f cuda-vectoradd.yaml
sleep 25
kubectl logs cuda-vectoradd             # expect: Test PASSED
kubectl delete -f cuda-vectoradd.yaml

E1. Restore the workload

kubectl apply -f gpu-burn.yaml
kubectl get pods

E2. Inference with replicas + PodDisruptionBudget (why production has many nodes)

Free the GPU first so the inference pod can schedule:

kubectl delete -f gpu-burn.yaml --ignore-not-found

Deploy an “inference” service with 2 replicas and a PDB that forbids dropping to zero:

cat > inference.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: inference
spec:
  replicas: 2
  selector:
    matchLabels: { app: inference }
  template:
    metadata:
      labels: { app: inference }
    spec:
      containers:
      - name: inference
        image: nvcr.io/nvidia/cuda:12.4.1-base-ubuntu22.04
        command: ["bash","-c","sleep infinity"]
        resources:
          limits:
            nvidia.com/gpu: 1
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: inference-pdb
spec:
  minAvailable: 1
  selector:
    matchLabels: { app: inference }
EOF

kubectl apply -f inference.yaml
kubectl get pods -l app=inference

See the PDB protect the service — try to drain and watch it get blocked:

kubectl drain $NODE --ignore-daemonsets --delete-emptydir-data
# Blocked: "Cannot evict pod ... would violate the pod's disruption budget"
# Press Ctrl+C to stop the attempt
kubectl uncordon $NODE

Simulation cleanup

kubectl delete -f gpu-burn.yaml --ignore-not-found
kubectl delete -f inference.yaml --ignore-not-found
kubectl delete pdb inference-pdb --ignore-not-found
kubectl uncordon $NODE 2>/dev/null
kubectl get pods