Cluster Setup

A typical Gaudi cluster looks like this:

• Compute nodes: bare-metal servers, 8 Gaudi accelerators each.
• Node-to-node networking: Gaudi chips have built-in RDMA networking ports (RoCEv2), no separate InfiniBand card needed like on NVIDIA. Gaudi 3 can often connect node-to-node directly for small clusters; bigger clusters need external switches.
• Management network, storage, monitoring stack: identical to our NVIDIA setup, same tools, same team.
• Job scheduling: Slurm or Kubernetes, with Gaudi nodes kept in their own pool so jobs land on the right hardware.

                        +-------------------------+
                        |   Management Plane      |
                        |  (BCM, Prometheus,       |
                        |   Grafana, Alertmanager) |
                        +------------+------------+
                                     |
                         1/10GbE OOB Management Network
                                     |
        +----------------------------+----------------------------+
        |                            |                             |
  +-----v-----+               +-----v-----+                 +-----v-----+
  |  Gaudi     |               |  Gaudi     |                 |  Gaudi     |
  |  Node 1    |               |  Node 2    |      ...        |  Node N    |
  |  8x Gaudi  |               |  8x Gaudi  |                 |  8x Gaudi  |
  |  cards     |               |  cards     |                 |  cards     |
  +-----+-----+               +-----+-----+                 +-----+-----+
        |                            |                             |
        +----------------------------+----------------------------+
                                     |
                     RoCEv2 Fabric (onboard NICs; direct
                     mesh for small clusters, leaf-spine
                     Ethernet switches for scale-out)
                                     |
                        +------------v------------+
                        |     Shared Storage        |
                        |   (Lustre / NFS / WEKA)   |
                        +----------------------------+

• Gaudi 2: 24 RoCEv2 ports per card, 21 used for scale-out (node-to-node) and 3 for scale-up (intra-node all-to-all mesh between the 8 accelerators).
• Gaudi 3: same port layout, doubled per-port bandwidth (roughly 2x Gaudi 2 fabric throughput).
• Small clusters (typically up to 32 nodes / 256 cards) can often wire nodes directly card-to-card in a mesh without external switches, cutting hardware cost.
• Larger clusters need a leaf-spine Ethernet fabric with RoCEv2-capable switches (e.g. Arista, NVIDIA Spectrum, or similar 400GbE-class gear). Standard Ethernet, not a proprietary interconnect; this is the main operational difference from an InfiniBand-based NVIDIA fabric.
• Lossless Ethernet configuration (PFC/ECN) on the switches matters for RoCEv2 performance; treat this the same way you'd tune an NVIDIA RoCE fabric, since there's no InfiniBand subnet manager to hide congestion behavior.

• Host CPU/memory: sized like our NVIDIA hosts, plenty of cores and RAM to keep the 8 accelerators fed; no Gaudi-specific host requirement beyond driver/firmware compatibility (see Software & Firmware).
• Local NVMe: recommended per node for checkpoint/scratch I/O, same as NVIDIA nodes.
• Shared storage: reuse the existing parallel filesystem (Lustre/WEKA/etc.), no Gaudi-specific storage stack required.

• Tag Gaudi nodes with a distinct partition/label (Slurm partition or Kubernetes node label/taint) so standard NVIDIA jobs don't accidentally land on them and vice versa.
• HCCL (Habana Collective Communications Library) is the NCCL equivalent; set `HCCL_*` env vars analogous to how `NCCL_*` vars are tuned today, particularly around scale-out topology hints if nodes aren't in a simple mesh.
• Health checks: monitor for downed RoCE links and card resets the same way GPU health checks are wired into scheduler drain logic today.

← Previous | Guide Index | Next →