This is an old revision of the document!
Cluster Setup
A typical Gaudi cluster looks like this:
Compute nodes: bare-metal servers, 8 Gaudi accelerators each.
Node-to-node networking: Gaudi chips have built-in RDMA networking ports (RoCEv2) — no separate InfiniBand card needed like on NVIDIA. Gaudi 3 can often connect node-to-node directly for small clusters; bigger clusters need external switches.
Management network, storage, monitoring stack: identical to our NVIDIA setup — same tools, same team.
Job scheduling: Slurm or Kubernetes, with Gaudi nodes kept in their own pool so jobs land on the right hardware.
Networking Topology
Gaudi 2: 24 RoCEv2 ports per card — 21 used for scale-out (node-to-node) and 3 for scale-up (intra-node all-to-all mesh between the 8 accelerators).
Gaudi 3: same port layout, doubled per-port bandwidth (roughly 2x Gaudi 2 fabric throughput).
Small clusters (typically up to 32 nodes / 256 cards) can often wire nodes directly card-to-card in a mesh without external switches, cutting hardware cost.
Larger clusters need a leaf-spine Ethernet fabric with RoCEv2-capable switches (e.g. Arista, NVIDIA Spectrum, or similar 400GbE-class gear). Standard Ethernet, not a proprietary interconnect — this is the main operational difference from an InfiniBand-based NVIDIA fabric.
Lossless Ethernet configuration (PFC/ECN) on the switches matters for RoCEv2 performance — treat this the same way you'd tune an NVIDIA RoCE fabric, since there's no InfiniBand subnet manager to hide congestion behavior.
Host & Storage Considerations
Host CPU/memory: sized like our NVIDIA hosts — plenty of cores and RAM to keep the 8 accelerators fed; no Gaudi-specific host requirement beyond driver/firmware compatibility (see
Software & Firmware).
Local NVMe: recommended per node for checkpoint/scratch I/O, same as NVIDIA nodes.
Shared storage: reuse the existing parallel filesystem (Lustre/WEKA/etc.) — no Gaudi-specific storage stack required.
Job Scheduling Notes
Tag Gaudi nodes with a distinct partition/label (Slurm partition or Kubernetes node label/taint) so standard NVIDIA jobs don't accidentally land on them and vice versa.
HCCL (Habana Collective Communications Library) is the NCCL equivalent — set `HCCL_*` env vars analogous to how `NCCL_*` vars are tuned today, particularly around scale-out topology hints if nodes aren't in a simple mesh.
Health checks: monitor for downed RoCE links and card resets the same way GPU health checks are wired into scheduler drain logic today.
← Previous | Guide Index | Next →
