1. Prepare — record current versions, verify GPU health, stop all workloads
2. Validate Package — checksum the firmware file, preview what will change
3. Flash GPU Tray — update VBIOS, NVSwitch, EROT, FPGA via nvfwupd
4. Reboot — wait for background copy to complete, then cold reboot
5. Flash Motherboard Tray — update BMC, BIOS, PSUs; reset BMC
6. Validate — confirm new versions, GPU health, NVLink, smoke test

NOTE: Flashing the wrong firmware file or interrupting mid-flash can permanently brick a GPU. Read the full runbook before executing.

This runbook covers the end-to-end firmware upgrade process for NVIDIA data center GPUs. It is self-contained — do not consult external documentation mid-upgrade.

Firmware is low-level software embedded in the GPU hardware itself. It controls boot behaviour, power management, and hardware communication — below the OS driver layer.

Components updated in a full DGX/HGX firmware upgrade:

• Fix GPU stability bugs or uncorrectable memory errors
• Apply security patches to the hardware trust chain
• Meet minimum firmware requirements for new driver versions
• Enable features such as Confidential Computing or NVLink enhancements

# --- nvfwupd ---
# Show installed vs package versions
nvfwupd -t ip=<BMC_IP> user=<USER> password=<PASS> show_version -p <pkg.fwpkg>
 
# Update all firmware components
nvfwupd -t ip=<BMC_IP> user=<USER> password=<PASS> update_fw -p <pkg.fwpkg> -y
 
# Update specific components only (with selector JSON)
nvfwupd -t ip=<BMC_IP> user=<USER> password=<PASS> update_fw -p <pkg.fwpkg> -y -s selector.json
 
# Force downgrade controls
nvfwupd -t ip=<BMC_IP> user=<USER> password=<PASS> force_update enable
nvfwupd -t ip=<BMC_IP> user=<USER> password=<PASS> force_update status
nvfwupd -t ip=<BMC_IP> user=<USER> password=<PASS> force_update disable
 
# --- NVFlash (standalone PCIe GPUs only) ---
sudo nvflash --list                                        # List all GPUs and indices
sudo nvflash --version                                     # Show VBIOS version (GPU 0)
sudo nvflash --index=0 --save backup_$(date +%Y%m%d).rom  # Backup current VBIOS
sudo nvflash --index=0 new_firmware.rom                    # Flash new VBIOS to GPU 0
 
# --- nvidia-smi ---
nvidia-smi -L                                              # List all GPUs
nvidia-smi -q                                              # Full GPU info
nvidia-smi -q | grep "VBIOS Version"                      # VBIOS version only
nvidia-smi --query-compute-apps=pid,name,used_memory --format=csv  # Active GPU processes
nvidia-smi nvlink -s -i 0                                  # NVLink status for GPU 0
 
# --- ipmitool ---
ipmitool -I lanplus -H <BMC_IP> -U <USER> -P <PASS> chassis power cycle  # Cold reboot
ipmitool -I lanplus -H <BMC_IP> -U <USER> -P <PASS> mc reset cold        # BMC-only reset
ipmitool -I lanplus -H <BMC_IP> -U <USER> -P <PASS> mc info              # Confirm BMC online
ipmitool -I lanplus -H <BMC_IP> -U <USER> -P <PASS> sel list             # BMC event log
 
# --- nvsm ---
nvsm show health                                           # System-wide health (DGX only)

Complete every item below before starting. Do not proceed if any item cannot be confirmed.

• [ ] BMC IP address, username, and password (administrator privileges)
• [ ] SSH access to the target server
• [ ] Access to NVIDIA Enterprise Support Portal to download firmware packages

• [ ] Correct .fwpkg file downloaded for your specific GPU model
• [ ] SHA256 checksum verified against NVIDIA release notes
• [ ] Package stored on the machine that will run nvfwupd

• [ ] All GPU workloads stopped — training jobs, inference servers, Kubernetes GPU pods
• [ ] nvidia-smi is not running — even passive queries block VBIOS updates
• [ ] Maintenance window scheduled and all stakeholders notified (minimum 2 hours)
• [ ] VBIOS backed up using NVFlash (standalone PCIe GPUs only)

• [ ] Management host can ping the BMC IP
• [ ] Server is on stable power — no UPS-only or unreliable supply
• [ ] No other firmware operations are running on the same server
• [ ] Server reports healthy — confirmed via nvsm show health or BMC dashboard

• [ ] Current firmware versions recorded in your change ticket
• [ ] Previous firmware .fwpkg file saved or accessible
• [ ] Rollback procedure reviewed (Section 9)

# DGX / HGX systems
nvfwupd -t ip=<BMC_IP> user=<USER> password=<PASS> show_version
 
# Standalone PCIe GPU
nvidia-smi -q | grep "VBIOS Version"

Copy the full output to your change ticket. Fill in the table:

nvidia-smi -L       # All expected GPUs must appear
nvsm show health    # All components must show OK  (DGX only)

If any GPU is missing or any component is degraded — stop. Investigate before proceeding.

Stop all of the following before continuing:

• AI/ML training jobs, inference servers, GPU-accelerated applications
• Any monitoring script or agent that runs nvidia-smi on a schedule
• Kubernetes GPU pods — drain the node first:

kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data

Confirm no processes remain:

nvidia-smi --query-compute-apps=pid,name,used_memory --format=csv
# Output must be completely empty before continuing

Skip this step for DGX/HGX systems managed by nvfwupd.

sudo nvflash --index=0 --save gpu0_backup_$(date +%Y%m%d).rom

Store the .rom file on a separate host. This is your only rollback option if the flash fails.

sha256sum nvfw_DGX-HGX-H100-H200x8_<version>.fwpkg

Compare the output against the checksum in the NVIDIA release notes. If they do not match — re-download the package before proceeding.

nvfwupd -t ip=<BMC_IP> user=<USER> password=<PASS> show_version -p <pkg.fwpkg>

Every component with a version mismatch will be updated. Confirm this matches your intent before proceeding.

Do not close your terminal, disconnect from the BMC, or allow the server to lose power once flashing begins. Any interruption mid-flash can permanently damage hardware.

Updates VBIOS, NVSwitch, EROT, and FPGA firmware for all GPUs in the system.

nvfwupd -t ip=<BMC_IP> user=<USER> password=<PASS> \
  update_fw -p nvfw_DGX-HGX-H100-H200x8_<version>.fwpkg -y

To update specific components only, create a selector JSON and pass it with -s:

{ "Components": ["VBIOS"] }

nvfwupd -t ip=<BMC_IP> user=<USER> password=<PASS> \
  update_fw -p <pkg.fwpkg> -y -s selector.json

The firmware is written to the GPU's flash storage in the background. You must wait for this before rebooting.

curl -k -u <USER>:<PASS> \
  https://<BMC_IP>/redfish/v1/Systems/HGX_Baseboard_0/Oem/Nvidia/BackgroundCopyStatus

Poll every 60 seconds. Only proceed when you see:

"BackgroundCopyStatus": "Completed"

If you see Failed — do not reboot. See the Troubleshooting section.

ipmitool -I lanplus -H <BMC_IP> -U <USER> -P <PASS> chassis power cycle

Wait 5–10 minutes for the server to fully boot.

After the server is back online, update BMC, BIOS, and PSU firmware:

nvfwupd -t ip=<BMC_IP> user=<USER> password=<PASS> \
  update_fw -p nvfw_DGX_<version>.fwpkg -y -s update_bmc.json

The motherboard tray uses a different .fwpkg file than the GPU tray. Check the NVIDIA release notes for the correct package name for your system.

ipmitool -I lanplus -H <BMC_IP> -U <USER> -P <PASS> mc reset cold

Wait 2–3 minutes, then confirm the BMC is back online:

ipmitool -I lanplus -H <BMC_IP> -U <USER> -P <PASS> mc info

sudo rmmod nvidia_uvm
sudo rmmod nvidia_drm
sudo rmmod nvidia_modeset
sudo rmmod nvidia
 
# Confirm all modules are unloaded (output must be empty)
lsmod | grep nvidia

# Flash GPU at index 0 — increment --index for additional GPUs (1, 2, ...)
sudo nvflash --index=0 new_vbios_firmware.rom
# NVFlash will prompt — review current vs new version shown, then type "yes"

sudo reboot

# DGX / HGX
nvfwupd -t ip=<BMC_IP> user=<USER> password=<PASS> show_version
 
# Standalone PCIe
nvidia-smi -q | grep "VBIOS Version"

Every updated component must now show the new version.

nvidia-smi -L

All expected GPUs must appear with correct model names. No Unknown or error entries.

nvidia-smi -q        # Check per-GPU: VBIOS version, ECC mode, temperature, power draw
nvsm show health     # DGX only — all components must show OK

nvidia-smi nvlink -s -i 0

All NVLink ports must show Active. Any inactive port must be investigated before restoring workloads.

python3 -c "import torch; print(torch.cuda.is_available()); print(torch.cuda.get_device_name(0))"

Expected: True and the correct GPU model name.

Only proceed after all checks above pass:

kubectl uncordon <node-name>

• Restart inference services and training jobs
• Re-enable GPU monitoring scripts

Monitor the system for 30–60 minutes after restoring workloads.

Only downgrade if the new firmware causes a confirmed regression. Raise an NVIDIA Support case before attempting.

# Unload driver
sudo rmmod nvidia_uvm
sudo rmmod nvidia_drm
sudo rmmod nvidia_modeset
sudo rmmod nvidia
 
# Flash the saved backup
sudo nvflash --index=0 gpu0_backup_<date>.rom
 
# Reboot
sudo reboot

Then re-run all Phase 3 validation steps.

# Step 1 — Enable ForceUpdate flag (required to allow downgrade)
nvfwupd -t ip=<BMC_IP> user=<USER> password=<PASS> force_update enable
 
# Step 2 — Confirm flag is active
nvfwupd -t ip=<BMC_IP> user=<USER> password=<PASS> force_update status
 
# Step 3 — Flash the previous firmware version
nvfwupd -t ip=<BMC_IP> user=<USER> password=<PASS> \
  update_fw -p nvfw_DGX-HGX-H100-H200x8_<old_version>.fwpkg -y
 
# Step 4 — Wait for BackgroundCopyStatus: Completed, then cold reboot (same as Steps 2.2–2.3)
 
# Step 5 — MANDATORY: Disable ForceUpdate immediately after
nvfwupd -t ip=<BMC_IP> user=<USER> password=<PASS> force_update disable
 
# Step 6 — Confirm it is disabled
nvfwupd -t ip=<BMC_IP> user=<USER> password=<PASS> force_update status

Then re-run all Phase 3 validation steps.

• NVIDIA DGX H100/H200 Firmware Update Guide — primary reference: nvfwupd commands, update sequence, downgrade, troubleshooting
• NVIDIA DGX A100 Firmware Update Container Release Notes
• NVIDIA Enterprise Support Portal — firmware package downloads (login required)