• RDMA-capable NVIDIA network adapter — this is the non-negotiable one. Supported families:

• NVIDIA GPU Operator v25.3.x+ — only required if workloads need GPUDirect RDMA (GPU-to-NIC direct memory access for GPU-to-GPU networking). Not required for network-only use cases.
• Validated NVIDIA Data Center systems (IGX Orin, Grace ARM Server, DGX/HGX GB200 NVL72, DGX/HGX B200, RTX PRO 6000 Blackwell Server) each pair a specific CPU/GPU/NIC/OS combination — worth checking directly against the Platform Support page if targeting one of these.

The operator manages a containerized DOCA-OFED driver (the successor to MLNX_OFED) rather than requiring it to be preinstalled on the host — this is one of the operator's core value props. Two delivery paths:

• Precompiled driver container images — fastest node bring-up, no in-node compilation. GA only for generic-flavor Ubuntu 22.04/24.04 kernels; nvidia/aws/azure/oracle kernel flavors are Tech Preview with limited testing; images are currently unsigned.
• Custom precompiled container — required for any kernel/OS combination not covered above (e.g., a non-generic cloud kernel); built from NVIDIA-provided Dockerfiles and hosted in your own registry.

helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update
 
helm install network-operator nvidia/network-operator \
  -n nvidia-network-operator \
  --create-namespace \
  --version v25.10.0 \
  --set sriovNetworkOperator.enabled=true \
  --wait
 
kubectl -n nvidia-network-operator get pods

This installs the operator controller and CRDs only — no host is touched until a NicClusterPolicy is applied (Section 3).

For anything beyond quick-start defaults, fetch the chart and drive the install from a values.yaml instead of ad hoc –set flags:

helm fetch nvidia/network-operator --version v25.10.0 --untar
# edit values.yaml
helm install network-operator ./network-operator -n nvidia-network-operator \
  --create-namespace -f values.yaml

kubectl -n nvidia-network-operator get pods

All controller pods (network-operator, and any enabled sub-charts like sriov-network-operator) should reach Running. This confirms the control plane is up — it does not confirm anything about NIC configuration, which only happens once a NicClusterPolicy is applied.

NicClusterPolicy (mellanox.com/v1alpha1) is the single central custom resource the operator continuously reconciles against. It is effectively the declarative “desired state” for the entire host networking stack across the cluster: which driver version runs, which device plugins are active, which CNIs and IPAM are deployed, and how upgrades should behave. Everything downstream — SR-IOV networks, MacVLAN networks, IPoIB networks — depends on a healthy NicClusterPolicy first.

apiVersion: mellanox.com/v1alpha1
kind: NicClusterPolicy
metadata:
  name: nic-cluster-policy
  namespace: nvidia-network-operator
spec:
  ofedDriver:
    image: doca-driver
    repository: nvcr.io/nvidia/mellanox
    version: doca3.2.0-25.10-1.2.8.0-2
    upgradePolicy:
      autoUpgrade: true
      maxParallelUpgrades: 1
      safeLoad: false
      drain:
        enable: true
        force: false
        podSelector: ""
        deleteEmptyDir: true

kubectl get -n nvidia-network-operator nicclusterpolicies.mellanox.com nic-cluster-policy -o yaml

status:
  appliedStates:
  - name: state-multus-cni
    state: ready
  - name: state-OFED
    state: ready
  - name: state-RDMA-device-plugin
    state: ready
  - name: state-nv-ipam-cni
    state: ready
  - name: state-SRIOV-device-plugin
    state: ignore   # not configured in this CR — not a failure
  state: ready

state: ready at the top level means the declared policy has been fully rolled out cluster-wide. A sub-state of ignore means that feature simply wasn't requested — not an error.

Any manual edit affecting device plugin configuration (e.g., NIC selectors) requires a manual device-plugin pod restart — this is a standing operator limitation, independent of whether the edit was part of a version upgrade or a plain reconfiguration.

• SR-IOV vs. Host Device — both give hardware-level performance, but SR-IOV slices one physical NIC into many isolated VFs (many tenants per node), while Host Device hands the entire physical NIC to a single pod (zero sharing).
• MacVLAN vs. IPoIB — MacVLAN is the Ethernet/RoCE-oriented secondary-network pattern; IPoIB is specifically for InfiniBand fabrics where ordinary IP addressing (rather than native IB addressing) is still wanted.
• SR-IOV InfiniBand — combines SR-IOV virtualization with native IB partitioning; typically the highest-performance option for large IB-based AI training clusters, and it's the one mode that depends on the IB Kubernetes plugin for pkey/GUID assignment.
• All “shared” modes (RDMA shared device, MacVLAN, IPoIB) ultimately compete for the same physical RDMA resources at the node level — the device-plugin resource counts configured in NicClusterPolicy (Section 3.2) become the real admission-control mechanism for how many pods can get RDMA on a given node.

Each mode has a ready-made NicClusterPolicy + secondary-network manifest pair in the Quick Start Guide (SR-IOV Network with RDMA, Host Device Network with RDMA, IPoIB with RDMA Shared Device, MacVLAN with RDMA Shared Device, SR-IOV InfiniBand Network with RDMA) — these are the fastest way to see a working reference for each pattern rather than building NicClusterPolicy fields from scratch.

helm upgrade only updates the operator's own controller and CRDs. It does not change the DOCA-OFED driver or any component version actually running on nodes — those live in NicClusterPolicy and must be bumped in a separate, manual step. This is the single most common source of “I upgraded but nothing changed” confusion.

Step 1 — Fetch the new chart:

helm fetch https:%%//%%helm.ngc.nvidia.com/nvidia/charts/network-operator-25.10.0.tgz
tar xf network-operator-25.10.0.tgz

Step 2 — Apply the Helm chart update (controller + CRDs only):

helm upgrade -n nvidia-network-operator network-operator nvidia/network-operator \
  --version=25.10.0 -f values-25.10.0.yaml --force

Step 3 — Manually update NicClusterPolicy with the new release's component versions (driver image tag, device plugin images, etc., from that release's component matrix), then re-apply:

kubectl apply -f nicclusterpolicy.yaml

Known limitation: a devicePlugin config change without a matching image bump may not auto-restart the plugin pod — manual kubectl delete pod restart may be required.

Step 4 — Driver rollout across nodes. Governed by ofedDriver.upgradePolicy:

• autoUpgrade — master on/off switch
• maxParallelUpgrades — throttles concurrent node upgrades (0 = unlimited)
• safeLoad — cordon/drain before the first containerized driver load, to prevent pods silently losing network interfaces mid-flight
• drain — enable/force/podSelector/timeoutSeconds/deleteEmptyDir

Each node walks a tracked state machine via the nvidia.com/ofed-driver-upgrade-state label: upgrade-required → cordon-required → wait-for-jobs-required → drain-required → pod-restart-required → uncordon-required → upgrade-done (or upgrade-failed, requiring manual intervention).

Two execution modes:

• In-place (default) — the operator's own internal scheduler performs cordon/drain/uncordon directly.
• Requestor mode — delegates those operations to the NVIDIA Maintenance Operator via NodeMaintenance objects (MAINTENANCE_OPERATOR_ENABLED=true, maintenanceOperator.enabled: true in Helm values). Useful if node maintenance is already standardized around that operator elsewhere in the cluster.

PodDisruptionBudget caveat: drains can silently fail if PDBs aren't accounted for — drain.podSelector must be scoped precisely to the pods that actually need to move. Over-broad selectors on clusters with strict PDBs (OpenShift especially) can block the whole drain.

Manual fallback (if autoUpgrade is off): the driver DaemonSet uses OnDelete update strategy, so a bare helm upgrade leaves the old driver pod running until it's explicitly deleted or the node reboots:

kubectl drain <NODE_NAME> --pod-selector=<SECONDARY_NETWORK_POD_SELECTOR>
kubectl delete pod -n nvidia-network-operator -l app=mofed-<OS_NAME>
kubectl uncordon <NODE_NAME>

  Do this per-node in production rather than all at once, to bound blast radius.

Step 5 — Verify:

kubectl get nodes -L nvidia.com/ofed-driver-upgrade-state
kubectl get -n nvidia-network-operator nicclusterpolicies.mellanox.com nic-cluster-policy -o yaml

Confirm every targeted node reads upgrade-done and the CR's overall status.state is ready. If a node sticks at upgrade-failed: drain it manually (kubectl drain –ignore-daemonsets), delete its driver pod, and — if Safe Driver Loading was enabled — remove the nvidia.com/ofed-driver-upgrade.driver-wait-for-safe-load node annotation to unblock the reload.

Calendar versioning: YY.MM.PP. When a new major (YY.MM) ships, the prior major moves to maintenance (patch/CVE fixes only); the one before that goes EOL. At time of writing: 25.10.x = GA, 25.7.x = Maintenance, 25.4.x and earlier = EOL. Plan upgrade cadence against this rather than waiting for forced EOL migration.

kubectl delete -n nvidia-network-operator nicclusterpolicies.mellanox.com nic-cluster-policy
helm uninstall network-operator -n nvidia-network-operator
kubectl get nicclusterpolicies.mellanox.com # expect "No resources found"

On OpenShift, OLM uninstall does not remove CRDs/CRs automatically — delete them explicitly afterward (oc delete crds hostdevicenetworks.mellanox.com macvlannetworks.mellanox.com nicclusterpolicies.mellanox.com).

Practical conclusion: the mechanical/API layer — Helm install, CRD structure, editing NicClusterPolicy, reading status, running upgrade commands — is fully hardware-agnostic and can be exercised anywhere with a Kubernetes API server. What cannot be validated without real ConnectX/BlueField hardware is anything downstream of NIC discovery: driver loading against real silicon, VF creation, RDMA device allocation, and actual throughput/latency behavior. Local environments are useful for validating “does my NicClusterPolicy YAML apply and pass schema/reconciliation cleanly” — everything hardware-dependent should be assessed against NVIDIA's documented, validated hardware/OS matrix (Section 1) instead.

Reference links

• Quick Start Guide: https://docs.nvidia.com/networking/display/kubernetes25100/quick-start/quick-start-k8s.html
• Life Cycle Management: https://docs.nvidia.com/networking/display/kubernetes25100/life-cycle-management.html
• Platform Support: https://docs.nvidia.com/networking/display/kubernetes25100/platform-support.html
• Customization Options & CRD Reference: https://docs.nvidia.com/networking/display/kubernetes25100/customizations/crds.html