Table of Contents

Keeping Software and Firmware Current

Rule of thumb: check Intel's Habana Support Matrix once a month for new releases. Validate anything new in a test environment before touching production.

Installing the driver

wget -qO - https://vault.habana.ai/artifactory/api/gpg/key/public | sudo apt-key add -
echo "deb https://vault.habana.ai/artifactory/debian focal main" | sudo tee /etc/apt/sources.list.d/habana.list
sudo apt-get update
sudo apt-get install -y habanalabs-dkms habanalabs-firmware habanalabs-container-toolkit
hl-smi   # confirms it worked

Checking installed versions

hl-smi -Q firmware-version -f csv,noheader   # firmware per card
dpkg -l | grep habanalabs                    # driver/package versions
hl-smi -Q driver_version -f csv,noheader     # driver version
modinfo habanalabs | grep version            # kernel module version

Rolling out an update, safely

  1. Tell the customer at least 48 hours ahead of time.
  2. Test the new version on one spare/lab node first.
  3. Write down the current firmware/driver versions before touching anything (hl-smi -Q firmware-version -f csv,noheader).
  4. Update one node per rack, run a quick validation (habana-validate + a short training job), then move to the next rack. Never update the whole cluster at once.
  5. Keep the old driver/firmware files around for 90 days in case we need to roll back — Intel's public downloads aren't always reliable to re-fetch later.
  6. After updating, confirm the driver loads and reports the version you expect.

Rolling back a bad update

  1. Stop any jobs running on the node and drain it from the scheduler first.
  2. Reinstall the previously saved package versions: sudo apt-get install –allow-downgrades habanalabs-dkms=<old-version> habanalabs-firmware=<old-version>.
  3. Reboot the node so the older kernel module and firmware load cleanly.
  4. Re-run hl-smi and habana-validate to confirm the rollback took and the card is healthy before returning the node to the pool.
  5. Note the rollback (and the reason) in the change log so the next update attempt doesn't repeat the same issue.

Troubleshooting a failed install

  1. hl-smi returns no devices: check dmesg | grep habanalabs for module load errors; confirm the DKMS build matches the running kernel (dkms status).
  2. Driver installs but firmware mismatch warning appears: re-run the firmware package install and reboot — a partial install is more common than actual corruption.
  3. Package install fails due to repo signature errors: the apt-key step may need re-running after an OS upgrade; re-add the GPG key before retrying.
  4. After a kernel upgrade, the DKMS module fails to rebuild automatically: run sudo dkms autoinstall manually, then reboot.

← Previous | Guide Index | Next →