====== Keeping Software and Firmware Current ======

**Rule of thumb:** check Intel's Habana Support Matrix once a month for new releases. Validate anything new in a test environment before touching production.

==== Installing the driver ====

<code bash>
wget -qO - https://vault.habana.ai/artifactory/api/gpg/key/public | sudo apt-key add -
echo "deb https://vault.habana.ai/artifactory/debian focal main" | sudo tee /etc/apt/sources.list.d/habana.list
sudo apt-get update
sudo apt-get install -y habanalabs-dkms habanalabs-firmware habanalabs-container-toolkit
hl-smi   # confirms it worked
</code>

==== Checking installed versions ====

<code bash>
hl-smi -Q firmware-version -f csv,noheader   # firmware per card
dpkg -l | grep habanalabs                    # driver/package versions
hl-smi -Q driver_version -f csv,noheader     # driver version
modinfo habanalabs | grep version            # kernel module version
</code>

==== Rolling out an update, safely ====

  - Tell the customer at least 48 hours ahead of time.
  - Test the new version on one spare/lab node first.
  - Write down the current firmware/driver versions before touching anything (''hl-smi -Q firmware-version -f csv,noheader'').
  - Update one node per rack, run a quick validation (''habana-validate'' + a short training job), then move to the next rack. Never update the whole cluster at once.
  - Keep the old driver/firmware files around for 90 days in case we need to roll back — Intel's public downloads aren't always reliable to re-fetch later.
  - After updating, confirm the driver loads and reports the version you expect.

==== Rolling back a bad update ====

  - Stop any jobs running on the node and drain it from the scheduler first.
  - Reinstall the previously saved package versions: ''sudo apt-get install --allow-downgrades habanalabs-dkms=<old-version> habanalabs-firmware=<old-version>''.
  - Reboot the node so the older kernel module and firmware load cleanly.
  - Re-run ''hl-smi'' and ''habana-validate'' to confirm the rollback took and the card is healthy before returning the node to the pool.
  - Note the rollback (and the reason) in the change log so the next update attempt doesn't repeat the same issue.

==== Troubleshooting a failed install ====

  - ''hl-smi'' returns no devices: check ''dmesg | grep habanalabs'' for module load errors; confirm the DKMS build matches the running kernel (''dkms status'').
  - Driver installs but firmware mismatch warning appears: re-run the firmware package install and reboot — a partial install is more common than actual corruption.
  - Package install fails due to repo signature errors: the apt-key step may need re-running after an OS upgrade; re-add the GPG key before retrying.
  - After a kernel upgrade, the DKMS module fails to rebuild automatically: run ''sudo dkms autoinstall'' manually, then reboot.

----

[[gaudi-cluster-setup|← Previous]] | [[gaudi-guide|Guide Index]] | [[gaudi-os-patching-hardening|Next →]]