| Both sides previous revisionPrevious revision | |
| wiki:ai:gaudi-software-firmware [2026/07/21 17:49] – swilson | wiki:ai:gaudi-software-firmware [2026/07/21 17:51] (current) – swilson |
|---|
| ====== Running Shared Clusters (Multi-Tenancy) ====== | ====== Keeping Software and Firmware Current ====== |
| |
| Many Gaudi clusters host more than one team or customer workload side by side, rather than being dedicated to a single customer. This page covers how we keep those tenants from interfering with each other, at both the scheduler level and the network level. | **Rule of thumb:** check Intel's Habana Support Matrix once a month for new releases. Validate anything new in a test environment before touching production. |
| |
| ===== Why This Matters ===== | ==== Installing the driver ==== |
| |
| On a shared cluster, one team's misbehaving job can, if left unchecked, starve every other team of compute or network bandwidth. The goal of everything below is to make that structurally impossible rather than relying on tenants to behave well. | <code bash> |
| | wget -qO - https://vault.habana.ai/artifactory/api/gpg/key/public | sudo apt-key add - |
| | echo "deb https://vault.habana.ai/artifactory/debian focal main" | sudo tee /etc/apt/sources.list.d/habana.list |
| | sudo apt-get update |
| | sudo apt-get install -y habanalabs-dkms habanalabs-firmware habanalabs-container-toolkit |
| | hl-smi # confirms it worked |
| | </code> |
| |
| ===== Isolation on Kubernetes ===== | ==== Checking installed versions ==== |
| |
| * **Namespaces:** each tenant is given its own Kubernetes namespace. This is the basic unit of separation; resources, secrets, and configuration in one tenant's namespace are not visible to another tenant by default. | <code bash> |
| * **Resource quotas:** a ''ResourceQuota'' object on each namespace caps how many Gaudi accelerators (via the ''habana.ai/gaudi'' resource) that tenant can claim at once. Without this, a single tenant could request every accelerator in the cluster and starve everyone else. | hl-smi -Q firmware-version -f csv,noheader # firmware per card |
| * **LimitRange:** sets sane defaults and maximums for individual pod requests, so one misconfigured job can't accidentally request an unreasonable amount of resource inside the quota. | dpkg -l | grep habanalabs # driver/package versions |
| * **Network policies:** ''NetworkPolicy'' objects block pod-to-pod traffic across namespaces by default. Cross-tenant traffic is only allowed if explicitly opened up, which we do not do unless the SOW calls for shared services between tenants. | hl-smi -Q driver_version -f csv,noheader # driver version |
| * **RBAC:** access to each namespace is scoped through Kubernetes role-based access control, so a tenant's credentials only grant permissions inside their own namespace, not cluster-wide. | modinfo habanalabs | grep version # kernel module version |
| | </code> |
| |
| ===== Isolation on Slurm ===== | ==== Rolling out an update, safely ==== |
| |
| * **Accounts:** each tenant gets a distinct Slurm account, which is how usage and fair-share accounting is tracked per tenant. | - Tell the customer at least 48 hours ahead of time. |
| * **Quality of Service (QOS) policies:** a QOS policy caps how much of the cluster (nodes, cores, GPU/accelerator count, wall-clock time) a given account can consume concurrently. This is the Slurm equivalent of a Kubernetes ResourceQuota. | - Test the new version on one spare/lab node first. |
| * **Partitions:** Gaudi nodes are typically kept in their own partition, separate from any NVIDIA partition in a mixed fleet, so jobs land on the accelerator type they were actually written for. | - Write down the current firmware/driver versions before touching anything (''hl-smi -Q firmware-version -f csv,noheader''). |
| * **Fair-share scheduling:** Slurm's fair-share algorithm de-prioritizes a tenant's future jobs if they've been consuming a disproportionate share of the cluster recently, which prevents one heavy user from permanently crowding out others even within their allowed quota. | - Update one node per rack, run a quick validation (''habana-validate'' + a short training job), then move to the next rack. Never update the whole cluster at once. |
| | - Keep the old driver/firmware files around for 90 days in case we need to roll back — Intel's public downloads aren't always reliable to re-fetch later. |
| | - After updating, confirm the driver loads and reports the version you expect. |
| |
| ===== Network Fairness on Shared Fabric ===== | ==== Rolling back a bad update ==== |
| |
| Kubernetes namespaces and Slurm QOS policies control compute access, but on a cluster with a shared RoCEv2 fabric, one tenant's job can still flood the network and slow everyone else's distributed training down, even if their compute usage is within quota. | - Stop any jobs running on the node and drain it from the scheduler first. |
| | - Reinstall the previously saved package versions: ''sudo apt-get install --allow-downgrades habanalabs-dkms=<old-version> habanalabs-firmware=<old-version>''. |
| | - Reboot the node so the older kernel module and firmware load cleanly. |
| | - Re-run ''hl-smi'' and ''habana-validate'' to confirm the rollback took and the card is healthy before returning the node to the pool. |
| | - Note the rollback (and the reason) in the change log so the next update attempt doesn't repeat the same issue. |
| |
| * **Traffic-priority classes (QOS/PFC):** we apply Priority Flow Control classes on the switch fabric so no single tenant's traffic can monopolize bandwidth and starve another tenant's collective communication (allreduce, etc.) traffic. | ==== Troubleshooting a failed install ==== |
| * **Why this is separate from compute quotas:** a tenant staying within their accelerator quota can still generate a disproportionate amount of network traffic per accelerator, depending on their model's communication pattern. Network fairness has to be enforced independently of compute fairness for this reason. | |
| |
| ===== Routine Checks ===== | - ''hl-smi'' returns no devices: check ''dmesg | grep habanalabs'' for module load errors; confirm the DKMS build matches the running kernel (''dkms status''). |
| | - Driver installs but firmware mismatch warning appears: re-run the firmware package install and reboot — a partial install is more common than actual corruption. |
| ^ How Often ^ What We Check ^ | - Package install fails due to repo signature errors: the apt-key step may need re-running after an OS upgrade; re-add the GPG key before retrying. |
| | Daily | ''kubectl get nodes'' / ''sinfo'' for any node stuck in an unexpected drain or cordon state. | | - After a kernel upgrade, the DKMS module fails to rebuild automatically: run ''sudo dkms autoinstall'' manually, then reboot. |
| | Weekly | Device-plugin and control-plane high-availability verification (confirming the scheduler itself has no single point of failure). | | |
| | Monthly | Version and patch compliance audit across the device plugin, control plane, and worker nodes. | | |
| | |
| A node stuck in an unexpected drained or cordoned state is a common early symptom of a tenant-isolation problem: it usually means a job failed to release its resources cleanly, and left the node unavailable for the next tenant's workload. | |
| | |
| ===== What Happens When Isolation Fails ===== | |
| | |
| If a tenant does manage to exceed its allotted share, either through a quota misconfiguration or, more rarely, an actual scheduler bug, this is handled as a standard incident (see [[gaudi-incident-change-management|Handling Incidents and Making Changes]]), not silently absorbed. The affected tenant's workload is not automatically throttled retroactively; we investigate root cause, fix the quota/policy gap, and document it, since a recurring quota bypass is a configuration-drift issue (see [[gaudi-configuration-consistency|Keeping Configuration Consistent]]). | |
| | |
| ===== Related Pages ===== | |
| | |
| * [[gaudi-configuration-consistency|Keeping Configuration Consistent]] — how we catch quota/policy drift before it becomes a real problem. | |
| * [[gaudi-incident-change-management|Handling Incidents and Making Changes]] — what happens when isolation actually fails. | |
| * [[gaudi-cluster-setup|Cluster Setup]] — where the RoCEv2 fabric this page's network fairness section depends on is described. | |
| |
| ---- | ---- |
| |
| [[gaudi-configuration-consistency|← Previous]] | [[gaudi-guide|Guide Index]] | [[gaudi-incident-change-management|Next →]] | [[gaudi-cluster-setup|← Previous]] | [[gaudi-guide|Guide Index]] | [[gaudi-os-patching-hardening|Next →]] |
| |
