1. Prepare — record current versions, verify GPU health, stop all workloads
2. Validate Package — checksum the firmware file, preview what will change
3. Phase 2 (in this exact order):
   1. BMC firmware update
   2. BMC reset (mc reset cold)
   3. Motherboard tray update (SBIOS, CPLD, PSU, etc.)
   4. GPU tray update (VBIOS, NVSwitch, EROT, FPGA)
   5. Poll BackgroundCopyStatus until Completed
   6. Chassis power cycle (graceful, via Redfish/ipmitool)
   7. Re-check versions on staged components
   8. CommitImage on HostBMC_0 and HostBIOS_0
   9. AC power cycle (full PDU/cord removal, 5–6 min)
   10. NIC firmware via mlxfwmanager on the host OS; TPM, NVMe via BMC (last — independent of the EROT/CPLD chain)
4. Validate — confirm new versions, GPU health, NVLink, smoke test

<note warning> Flashing the wrong firmware file or interrupting mid-flash can permanently brick a GPU. Read the full runbook before executing.

Critical ordering: BMC must be updated and reset *before* the motherboard and GPU trays so the BMC speaks the new Redfish schema used by the rest of the bundle. Skipping the AC power cycle leaves EROT/CPLD images staged but not activated. </note>

This runbook covers the end-to-end firmware upgrade process for NVIDIA DGX/HGX H100/H200 systems, GB200/B200 (Blackwell) systems, and standalone PCIe GPUs.

Scope — what this runbook covers:

• Hopper: DGX H100/H200, HGX H100/H200
• Blackwell: DGX B200, HGX B200, GB200 NVL72 compute tray (per-tray BMC), GB300
• Standalone PCIe GPUs via NVFlash
• Components: VBIOS, NVSwitch, EROT, FPGA, BMC, SBIOS, CPLD, PSU, NIC, TPM, NVMe, and (Blackwell only) Grace CPU FW, NVLink Switch tray FW, ConnectX-7/8 onboard NICs

Blackwell-specific notes (GB200 / B200):

• GB200 NVL72 racks are driven from the NVIDIA Mission Control rack-firmware tooling. Refer to the Mission Control rack-firmware document shipped with the bundle — section 7.1B below summarises the flow but does not replace it.
• Bundles arrive as two NVIDIA-released packages plus the NVLink Switch tray package: nvfw_BMC_<P-number>.fwpkg and nvfw_Compute_<P-number>.fwpkg (Mission Control names them by NVIDIA “P-number” — e.g., P4972 = early Blackwell baseline, P4975 = first GA refresh, P4978 = current — and selector JSONs are named BMC_Full.json and Compute_Full.json).
• Ordering, simplified: chassis BMC (BMC_Full) → AC cycle between BMC and HMC updates → HMC + GPU/Grace complex (Compute_Full) → NVLink Switch tray → final rack-level AC cycle for EROT/CPLD. The AC cycle in the middle is required so the new chassis BMC re-enumerates the HMC under the updated Redfish schema before Compute_Full is pushed.
• Activation: Mission Control uses the documented activation method activate_fw -c RF_AUX_PWR_CYCLE (auxiliary-rail power cycle via Redfish), which replaces the ad-hoc “chassis power cycle” step that Hopper uses.
• The final mandatory rack-level AC cycle for EROT/CPLD still applies — for GB200 NVL72 this means power-cycling the full rack PDU feeds (coordinate with DC ops).
• nvfwupd floor: 2.0.4 for Hopper; 2.0.5 for Blackwell (HMC + Grace + NVLink Switch tray Redfish paths), or 2.0.9 if you intend to run parallel multi-package updates across the rack.

Scope — what this runbook does NOT cover:

• DGX A100 (use the Firmware Update Container — separate runbook)
• Standalone Mellanox/ConnectX adapter firmware in non-DGX hosts (use mlxfwmanager separately)
• Quantum/Spectrum InfiniBand or Ethernet switch firmware
• Storage controller / RAID HBA firmware

<note important> EROT and CPLD images are staged into a secondary flash region by the update, but only activate on a full AC power removal — a warm reboot or chassis power cycle is not sufficient. </note>

The minimum driver depends on the firmware package version, not the GPU architecture alone. Refer to the per-package release notes shipped with the .fwpkg you intend to install. Do not rely on generic architecture → driver tables; they go stale quickly as new packages add features and ABI requirements.

• NVSwitch chip — silicon on the GPU baseboard. Inventory path: /redfish/v1/UpdateService/FirmwareInventory/HGX_FW_NVSwitch_<n>. Updated via the compute tray BMC as part of the GPU tray package.
• NVLink Switch tray — separate rack-level switch chassis in GB200 NVL72 racks. Inventory path: /redfish/v1/UpdateService/FirmwareInventory/NVSwitch_Tray_*. Updated via the NVLink Switch tray BMC, not the compute tray BMC, with its own .fwpkg.

Selecting the wrong target will return a Redfish 404 and silently skip the component.

Schedule a 3-hour window for Hopper, 4-hour window for Blackwell B200, 5-hour window for GB200 NVL72.

• Hopper (H100/H200) minimum: nvfwupd 2.0.4 — earlier versions do not support CommitImage or the inventory paths used below.
• Blackwell (B200 / GB200 / GB300) minimum: nvfwupd 2.0.5 — required for HMC, Grace CPU, and NVLink Switch tray Redfish paths. Use 2.0.9 if you plan to push multiple packages in parallel from the rack management host (BMC_Full + Compute_Full).
• Confirm with: nvfwupd –version
• Download from the NVIDIA Enterprise Support Portal.

# --- nvfwupd ---
nvfwupd --version                                          # MUST be >= 2.0.4
 
nvfwupd -t ip=<BMC_IP> user=<USER> password=<PASS> show_version
nvfwupd -t ip=<BMC_IP> user=<USER> password=<PASS> show_version -p <pkg.fwpkg>
 
nvfwupd -t ip=<BMC_IP> user=<USER> password=<PASS> update_fw -p <pkg.fwpkg> -y
nvfwupd -t ip=<BMC_IP> user=<USER> password=<PASS> update_fw -p <pkg.fwpkg> -y -s selector.json
 
nvfwupd -t ip=<BMC_IP> user=<USER> password=<PASS> force_update enable
nvfwupd -t ip=<BMC_IP> user=<USER> password=<PASS> force_update status
nvfwupd -t ip=<BMC_IP> user=<USER> password=<PASS> force_update disable
 
# Blackwell / Mission Control activation (replaces ipmitool mc reset cold)
nvfwupd -t ip=<BMC_IP> user=<USER> password=<PASS> servertype=GB200 \
  activate_fw -c RF_AUX_PWR_CYCLE
 
# --- mlxfwmanager (host OS, Hopper NIC updates) ---
sudo mlxfwmanager --query
sudo mlxfwmanager -u -i fw-ConnectX7-rel-<version>.bin --yes
 
# --- NVFlash ---
sudo nvflash --list
sudo nvflash --version
sudo nvflash --index=0 --save backup_$(date +%Y%m%d).rom
sudo nvflash --index=0 new_firmware.rom
 
# --- nvidia-smi ---
nvidia-smi -L
nvidia-smi -q
nvidia-smi -q | grep "VBIOS Version"
nvidia-smi --query-compute-apps=pid,name,used_memory --format=csv
nvidia-smi nvlink -s -i 0
 
# --- ipmitool ---
ipmitool -I lanplus -H <BMC_IP> -U <USER> -P <PASS> chassis power cycle
ipmitool -I lanplus -H <BMC_IP> -U <USER> -P <PASS> mc reset cold
ipmitool -I lanplus -H <BMC_IP> -U <USER> -P <PASS> mc info
ipmitool -I lanplus -H <BMC_IP> -U <USER> -P <PASS> sel list
 
# --- nvsm ---
nvsm show health

<note important> Complete every item below before starting. Do not proceed if any item cannot be confirmed. </note>

• [ ] BMC IP address, username, and password (administrator privileges)
• [ ] SSH access to the target server
• [ ] Physical or remote-hands access for the AC power cycle step
• [ ] Access to NVIDIA Enterprise Support Portal

• [ ] nvfwupd ≥ 2.0.4 installed on management host
• [ ] ipmitool, curl, jq available on management host

• [ ] Correct .fwpkg for your specific platform (GPU tray + motherboard tray packages)
• [ ] SHA256 checksum verified against release notes
• [ ] Per-package release notes reviewed for driver minimums and known issues
• [ ] Previous .fwpkg saved for rollback

• [ ] If current BMC firmware is v1.0.0 or v1.1.1, you must stage to v1.1.3 first, reset BMC, and verify, before attempting v24.09.1 (or any later bundle). The full hop is v1.0.0 / v1.1.1 → v1.1.3 → v24.09.1. Skipping the intermediate hop requires a factory reset of the BMC to recover.
• [ ] Confirm current BMC version (pin to the BMC Redfish path, not a substring match):

    nvfwupd -t ip=<BMC_IP> user=<USER> password=<PASS> show_version \
      | grep -E '(^|\s)HostBMC_0\b'
 

• [ ] All GPU workloads stopped
• [ ] nvidia-smi is not running (passive queries block VBIOS updates)
• [ ] Maintenance window: 3h Hopper / 4h B200 / 5h GB200 NVL72 (see Time Budget in Section 2)
• [ ] VBIOS backed up via NVFlash (standalone PCIe only)
• [ ] MIG disabled on all GPUs: sudo nvidia-smi -mig 0 (some packages refuse to flash with MIG instances active)
• [ ] nvidia-fabricmanager stopped: sudo systemctl stop nvidia-fabricmanager (DGX/HGX only; prevents dirty shutdown during chassis cycle)
• [ ] Confidential Computing (CC) mode disabled if previously enabled — some .fwpkg versions refuse VBIOS flash while CC is on. Check the package release notes.
• [ ] PSU redundancy healthy: nvsm show health | grep -i psu (all PSUs OK; PSU update will refuse if redundancy is degraded)
• [ ] Driver minimum verified against this package's release notes: nvidia-smi –query-gpu=driver_version –format=csv,noheader ≥ the minimum listed in the .fwpkg release notes
• [ ] Pre-upgrade SEL captured: ipmitool -I lanplus -H <BMC_IP> -U <USER> -P <PASS> sel list > preupgrade_sel_$(date +%Y%m%d).log

• [ ] Management host can ping the BMC IP
• [ ] Server on stable power
• [ ] No other firmware operations running on the same server
• [ ] nvsm show health clean

• [ ] Current firmware versions recorded in change ticket
• [ ] Previous firmware .fwpkg saved
• [ ] Rollback procedure reviewed (Section 9)

# Pin the full pre-upgrade output to a file for the change ticket
nvfwupd -t ip=<BMC_IP> user=<USER> password=<PASS> show_version \
  | tee preupgrade_versions_$(date +%Y%m%d_%H%M).txt
 
# Standalone PCIe GPU
nvidia-smi -q | grep "VBIOS Version" | tee preupgrade_vbios_$(date +%Y%m%d).txt

Copy full output to your change ticket and fill in:

nvidia-smi -L
nvsm show health

kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data
nvidia-smi --query-compute-apps=pid,name,used_memory --format=csv
# Output must be empty

# Disable MIG on all GPUs (no-op if already disabled)
sudo nvidia-smi -mig 0
 
# Stop fabric manager (DGX/HGX/Blackwell) — prevents a dirty shutdown
# when the chassis is power-cycled in Phase 2.
sudo systemctl stop nvidia-fabricmanager
systemctl is-active nvidia-fabricmanager   # must report "inactive"
 
# Confidential Computing — disable if currently enabled and the package
# release notes require it. (Skip if your environment never uses CC mode.)
# nvidia-smi conf-compute -srs 0

sudo nvflash --index=0 --save gpu0_backup_$(date +%Y%m%d).rom

sha256sum nvfw_DGX-HGX-H100-H200x8_<version>.fwpkg

nvfwupd -t ip=<BMC_IP> user=<USER> password=<PASS> show_version -p <pkg.fwpkg>

<note warning> Strict ordering — do not deviate. BMC first, then motherboard tray, then GPU tray, then poll, then chassis power cycle, then CommitImage, then AC cycle, then NIC/TPM/NVMe. </note>

Use the documented one-shot BMC selector. Do not hand-enumerate child paths — let nvfwupd resolve them from the package.

cat > selector_bmc.json <<'EOF'
{
  "Targets": [
    "/redfish/v1/UpdateService/FirmwareInventory/HostBMC_0"
  ]
}
EOF
 
nvfwupd -t ip=<BMC_IP> user=<USER> password=<PASS> \
  update_fw -p nvfw_DGX_<version>.fwpkg -y -s selector_bmc.json

ipmitool -I lanplus -H <BMC_IP> -U <USER> -P <PASS> mc reset cold
sleep 180
ipmitool -I lanplus -H <BMC_IP> -U <USER> -P <PASS> mc info
 
# Pin the recheck to the BMC Redfish path (not a substring match on "bmc",
# which also matches HGX_FW_BMC_* aggregate entries on some packages).
nvfwupd -t ip=<BMC_IP> user=<USER> password=<PASS> show_version \
  | grep -E '(^|\s)HostBMC_0\b'

Confirm the BMC reports the new version before continuing.

The motherboard tray uses the empty selector ({}), which tells nvfwupd to apply every motherboard-side component the package declares. Do not hand-enumerate HostBIOS_0 / CPLDMB_0 / PSU_* — the path names vary by SKU and package, and a typo silently skips the component.

cat > selector_mb.json <<'EOF'
{}
EOF
 
nvfwupd -t ip=<BMC_IP> user=<USER> password=<PASS> \
  update_fw -p nvfw_DGX_<version>.fwpkg -y -s selector_mb.json

Cross-check beforehand with show_version -p <pkg.fwpkg> to confirm which components the package will touch.

The GPU tray uses the HGX_0 one-shot selector, which targets the entire GPU baseboard inventory in a single call. The BMC fans out internally to all GPUs, NVSwitch chips, EROT, and FPGA — do not enumerate per-GPU paths.

cat > selector_gpu.json <<'EOF'
{
  "Targets": [
    "/redfish/v1/UpdateService/FirmwareInventory/HGX_0"
  ]
}
EOF
 
nvfwupd -t ip=<BMC_IP> user=<USER> password=<PASS> \
  update_fw -p nvfw_DGX-HGX-H100-H200x8_<version>.fwpkg -y -s selector_gpu.json

Verify the exact child paths your platform exposes with nvfwupd … show_version on a real DGX H100/H200 before relying on inventory names elsewhere in this runbook — they can differ between H100 and H200 chassis SKUs.

curl -k -u <USER>:<PASS> \
  https://<BMC_IP>/redfish/v1/Chassis/HGX_ERoT_BMC_0

Poll every 60 seconds until BackgroundCopyStatus reports Completed. If Failed — do not proceed; see Troubleshooting.

ipmitool -I lanplus -H <BMC_IP> -U <USER> -P <PASS> chassis power cycle

Wait 5–10 minutes for the server to fully boot.

nvfwupd -t ip=<BMC_IP> user=<USER> password=<PASS> show_version

Confirm staged versions are visible on BMC, SBIOS, GPU tray components.

CommitImage promotes the staged image to the active slot. Required for BMC and SBIOS.

# Commit BMC
curl -k -u <USER>:<PASS> -X POST \
  -H "Content-Type: application/json" \
  -d '{"Targets":["/redfish/v1/UpdateService/FirmwareInventory/HostBMC_0"]}' \
  https://<BMC_IP>/redfish/v1/UpdateService/Actions/Oem/NvidiaUpdateService.CommitImage
 
# Commit SBIOS
curl -k -u <USER>:<PASS> -X POST \
  -H "Content-Type: application/json" \
  -d '{"Targets":["/redfish/v1/UpdateService/FirmwareInventory/HostBIOS_0"]}' \
  https://<BMC_IP>/redfish/v1/UpdateService/Actions/Oem/NvidiaUpdateService.CommitImage

<note warning> This step is non-negotiable for EROT and CPLD activation. A chassis power cycle is not sufficient. </note>

1. Power off the host via OS shutdown or ipmitool chassis power off.
2. Physically remove all AC power cords (or open the PDU breakers feeding the chassis).
3. Wait 5–6 minutes to allow standby rails to fully discharge.
4. Reconnect AC power and power the server back on.
5. Wait 5–10 minutes for full boot.

These components are independent of the GPU/EROT activation chain and are updated last.

Hopper (DGX/HGX H100/H200) — NICs via host-OS mlxfwmanager, not Redfish. On Hopper the ConnectX adapters are not exposed under the HGX BMC's firmware inventory; the BMC will return 404 / “no such target” for HGX_FW_NIC_* paths. Update them from the host OS instead:

# Inventory and current versions on the host OS
sudo mlxfwmanager --query
 
# Apply the bundled MFT firmware image (matches your ConnectX SKU)
sudo mlxfwmanager -u -i fw-ConnectX7-rel-<version>.bin --yes
 
# Re-query to confirm the new FW Version is present and "Status: Done"
sudo mlxfwmanager --query

A subsequent chassis power cycle (or PCIe FLR via mlxfwreset) is required to activate the new NIC firmware — do this after the AC cycle in Step 2.9.

TPM and NVMe remain BMC-driven on Hopper. Confirm the exact inventory paths on your chassis with nvfwupd … show_version before flashing — names like TPM_0 / NVMe_0 (without the HGX_FW_ prefix) appear on some packages.

# After confirming the actual inventory names on your chassis:
nvfwupd -t ip=<BMC_IP> user=<USER> password=<PASS> \
  show_version | grep -Ei '(tpm|nvme)'
 
# Then build a selector_tpm_nvme.json with the paths returned above and:
nvfwupd -t ip=<BMC_IP> user=<USER> password=<PASS> \
  update_fw -p nvfw_DGX_<version>.fwpkg -y -s selector_tpm_nvme.json

Blackwell (B200 / GB200) — also update onboard ConnectX-7/8 NICs:

cat > selector_periph_bw.json <<'EOF'
{
  "Targets": [
    "/redfish/v1/UpdateService/FirmwareInventory/HGX_FW_ConnectX_0",
    "/redfish/v1/UpdateService/FirmwareInventory/HGX_FW_ConnectX_1",
    "/redfish/v1/UpdateService/FirmwareInventory/HGX_FW_ConnectX_2",
    "/redfish/v1/UpdateService/FirmwareInventory/HGX_FW_ConnectX_3",
    "/redfish/v1/UpdateService/FirmwareInventory/HGX_FW_TPM_0",
    "/redfish/v1/UpdateService/FirmwareInventory/HGX_FW_NVMe_0"
  ]
}
EOF
 
nvfwupd -t ip=<BMC_IP> user=<USER> password=<PASS> servertype=GB200 \
  update_fw -p nvfw_GB200_<version>.fwpkg -y -s selector_periph_bw.json

Then host reboot:

ipmitool -I lanplus -H <BMC_IP> -U <USER> -P <PASS> chassis power cycle

<note important> This subsection summarises the NVIDIA Mission Control rack-firmware document that ships with the Blackwell bundle. Cross-check every selector path and package P-number against the Mission Control doc for your bundle version before executing. The component path names below are the documented Mission Control names — they are not the same as the Hopper HGX_FW_* paths and must not be cross-pollinated. </note>

Blackwell delivers two top-level packages plus the NVLink Switch tray package:

Ordering (per Mission Control):

1. B1. Push BMC_Full to the chassis BMC.
2. B2. activate_fw -c RF_AUX_PWR_CYCLE on the chassis BMC.
3. B3. AC power cycle between BMC and HMC updates (rack PDU cycle for NVL72; chassis cord pull for standalone B200). Wait 5 min for standby drain. This is required so the new chassis BMC re-enumerates the HMC under the updated Redfish schema before Compute_Full is pushed.
4. B4. Push Compute_Full (HMC + Grace CPU FW + GPU/NVSwitch/EROT/FPGA on the compute tray).
5. B5. activate_fw -c RF_AUX_PWR_CYCLE for the Compute_Full payload.
6. B6. Push the NVLink Switch tray package from the rack-level management host against the NVLink Switch tray BMC (not the compute tray BMC).
7. B7. Final rack-level AC cycle (Step 2.9 / 2.9b) to activate EROT and CPLD on every tray.

The Mission Control selector JSON is a one-shot; do not enumerate child components.

cat > BMC_Full.json <<'EOF'
{
  "Targets": [
    "/redfish/v1/UpdateService/FirmwareInventory/HostBMC_0"
  ]
}
EOF
 
nvfwupd -t ip=<BMC_IP> user=<USER> password=<PASS> servertype=GB200 \
  update_fw -p nvfw_BMC_P4978.fwpkg -y -s BMC_Full.json

Blackwell uses the documented Mission Control activation verb, not ipmitool mc reset cold:

nvfwupd -t ip=<BMC_IP> user=<USER> password=<PASS> servertype=GB200 \
  activate_fw -c RF_AUX_PWR_CYCLE

This is not the final EROT/CPLD AC cycle — it sits between the two halves of the bundle and is mandatory.

1. Standalone B200: chassis cord pull, 5 min drain.
2. GB200 NVL72: rack-level PDU cycle (both A-side and B-side feeds), 10 min drain, 15 min boot.
3. Confirm the chassis BMC comes back on the new firmware before proceeding: nvfwupd … show_version | grep -E '(^|\s)HostBMC_0\b'.

Compute_Full bundles HMC, Grace CPU FW, GPU/NVSwitch/EROT/FPGA for the compute tray.

cat > Compute_Full.json <<'EOF'
{
  "Targets": [
    "/redfish/v1/UpdateService/FirmwareInventory/HGX_0"
  ]
}
EOF
 
nvfwupd -t ip=<BMC_IP> user=<USER> password=<PASS> servertype=GB200 \
  update_fw -p nvfw_Compute_P4978.fwpkg -y -s Compute_Full.json

nvfwupd -t ip=<BMC_IP> user=<USER> password=<PASS> servertype=GB200 \
  activate_fw -c RF_AUX_PWR_CYCLE

Run from the rack-level management host against the NVLink Switch tray BMC. Confirm the exact tray-side selector path names from the Mission Control NVLink-Switch addendum for your bundle — they have changed between P-numbers.

nvfwupd -t ip=<NVL_SW_BMC_IP> user=<USER> password=<PASS> servertype=NVL_SWITCH \
  update_fw -p nvfw_NVLSwitch_P4978.fwpkg -y -s NVLSwitch_Full.json
 
nvfwupd -t ip=<NVL_SW_BMC_IP> user=<USER> password=<PASS> servertype=NVL_SWITCH \
  activate_fw -c RF_AUX_PWR_CYCLE

For GB200 NVL72 the final AC step is a rack-level PDU cycle, not a single chassis cord pull. Coordinate with DC operations:

1. Schedule a full rack power-down window.
2. Open both A-side and B-side PDU feeds (the rack is fed redundantly — pulling only one side leaves standby rails energised).
3. Wait 10 minutes (longer than the 5–6 min for standalone DGX due to BBU/standby capacitance).
4. Restore power; allow 15 minutes for the full rack to come up before validation.

sudo rmmod nvidia_uvm nvidia_drm nvidia_modeset nvidia
lsmod | grep nvidia

sudo nvflash --index=0 new_vbios_firmware.rom

sudo reboot

nvfwupd -t ip=<BMC_IP> user=<USER> password=<PASS> show_version
nvidia-smi -q | grep "VBIOS Version"

Every component must report the new version. Pay particular attention to EROT and CPLD — if either still reports the old version, the AC power cycle was not effective; repeat Step 2.9.

nvidia-smi -L

nvidia-smi -q
nvsm show health

nvidia-smi nvlink -s -i 0

python3 -c "import torch; print(torch.cuda.is_available()); print(torch.cuda.get_device_name(0))"

# Diff against the pre-upgrade SEL captured in Section 4
ipmitool -I lanplus -H <BMC_IP> -U <USER> -P <PASS> sel list \
  > postupgrade_sel_$(date +%Y%m%d).log
diff preupgrade_sel_*.log postupgrade_sel_*.log
 
# Restart fabric manager and confirm healthy
sudo systemctl start nvidia-fabricmanager
sudo systemctl status nvidia-fabricmanager   # must be active (running)
 
# Re-enable MIG only if your workload requires it
# sudo nvidia-smi -mig 1

Investigate any new SEL entries (assertion events, thermal trips, fan failures) before restoring workloads.

kubectl uncordon <node-name>

Monitor for 30–60 minutes.

Same flow as the forward upgrade, but with the previous .fwpkg and force_update enable set first. Disable force_update immediately after. EROT/CPLD downgrades also require the AC cycle.

If the BMC is unreachable after an update (no Redfish, no IPMI, no SSH), do not attempt field recovery from a USB image. The BMC recovery image, layout, and procedure are NVIDIA-internal and SKU-specific; running the wrong image can permanently brick the chassis BMC and is not covered by warranty.

Instead:

1. Open an NVIDIA Enterprise Support case with severity matching your maintenance window.
2. Capture the BMC serial console at 115200 8N1 (rear-panel serial port or serial-over-USB on the management cable) from the moment the BMC fails through any attempted recovery. Attach the full log to the support case — NVIDIA will require it before authorising any recovery image.
3. Provide: chassis SKU, current BMC version (last known good), target BMC version, the package P-number / file name attempted, and confirmation of whether the v1.0.0 / v1.1.1 → v1.1.3 → v24.09.1 hop was followed.
4. Do not pull the chassis from the rack or attempt a factory reset until NVIDIA confirms the recovery path — some failures are recoverable from the host side without touching the BMC.

• NVIDIA DGX H100/H200 Firmware Update Guide
• Per-package release notes — always consult for driver minimums and known issues
• NVIDIA Enterprise Support Portal