When a Gaudi node is added to a cluster, it goes through the same repeatable process every time:

1. Physical setup: rack, cable, and power the node; connect it to both the management network and the RoCEv2 fabric.
2. BIOS/firmware config: set PXE boot order and enable IOMMU/SR-IOV/NUMA interleaving per the Gaudi profile.
3. Register it: add the node's MAC address and IPMI details to our provisioning system (BCM) under the gaudi2-ubuntu22 or gaudi3-ubuntu22 profile.
4. Image it: PXE-boot the base OS image, then run our configuration management playbook (installs monitoring agents, hardening baseline from OS Patching and Hardening, standard packages).
5. Install the Gaudi driver stack: as in Keeping Software and Firmware Current.
6. Validate: run habana-validate to confirm the card and fabric are healthy.
7. Add to the scheduler: register the node in the Slurm/Kubernetes Gaudi pool.
8. Log it: add the node to the asset register (serial number, location, firmware version, install date).

Taking a node out of service follows the same steps in reverse: drain it, confirm no jobs or data are still on it, revoke its access credentials, wipe its storage (see Hardware Replacement (RMA)), remove it from BCM, and update the asset register.

← Previous | Guide Index | Next →