Differences

This shows you the differences between two versions of the page.

--- wiki:ai:gaudi-software-firmware [2026/07/21 17:31] – created swilson
+++ wiki:ai:gaudi-software-firmware [2026/07/21 17:51] (current) – swilson
@@ Line 11: / Line 11: @@
 sudo apt-get install -y habanalabs-dkms habanalabs-firmware habanalabs-container-toolkit
 hl-smi   # confirms it worked
+</code>
+==== Checking installed versions ====
+<code bash>
+hl-smi -Q firmware-version -f csv,noheader   # firmware per card
+dpkg -l | grep habanalabs                    # driver/package versions
+hl-smi -Q driver_version -f csv,noheader     # driver version
+modinfo habanalabs | grep version            # kernel module version
 </code>
@@ Line 21: / Line 30: @@
   - Keep the old driver/firmware files around for 90 days in case we need to roll back — Intel's public downloads aren't always reliable to re-fetch later.
   - After updating, confirm the driver loads and reports the version you expect.
+==== Rolling back a bad update ====
+  - Stop any jobs running on the node and drain it from the scheduler first.
+  - Reinstall the previously saved package versions: ''sudo apt-get install --allow-downgrades habanalabs-dkms=<old-version> habanalabs-firmware=<old-version>''.
+  - Reboot the node so the older kernel module and firmware load cleanly.
+  - Re-run ''hl-smi'' and ''habana-validate'' to confirm the rollback took and the card is healthy before returning the node to the pool.
+  - Note the rollback (and the reason) in the change log so the next update attempt doesn't repeat the same issue.
+==== Troubleshooting a failed install ====
+  - ''hl-smi'' returns no devices: check ''dmesg | grep habanalabs'' for module load errors; confirm the DKMS build matches the running kernel (''dkms status'').
+  - Driver installs but firmware mismatch warning appears: re-run the firmware package install and reboot — a partial install is more common than actual corruption.
+  - Package install fails due to repo signature errors: the apt-key step may need re-running after an OS upgrade; re-add the GPG key before retrying.
+  - After a kernel upgrade, the DKMS module fails to rebuild automatically: run ''sudo dkms autoinstall'' manually, then reboot.
 ----

Combined Cloud Managed Services

Site Tools

Differences

Page Tools