====== Cluster Setup ======

A typical Gaudi cluster looks like this:

  * **Compute nodes:** bare-metal servers, 8 Gaudi accelerators each.
  * **Node-to-node networking:** Gaudi chips have built-in RDMA networking ports (RoCEv2), no separate InfiniBand card needed like on NVIDIA. Gaudi 3 can often connect node-to-node directly for small clusters; bigger clusters need external switches.
  * **Management network, storage, monitoring stack:** identical to our NVIDIA setup, same tools, same team.
  * **Job scheduling:** Slurm or Kubernetes, with Gaudi nodes kept in their own pool so jobs land on the right hardware.

===== Architecture Diagram =====

<code>
                        +-------------------------+
                        |   Management Plane      |
                        |  (BCM, Prometheus,       |
                        |   Grafana, Alertmanager) |
                        +------------+------------+
                                     |
                         1/10GbE OOB Management Network
                                     |
        +----------------------------+----------------------------+
        |                            |                             |
  +-----v-----+               +-----v-----+                 +-----v-----+
  |  Gaudi     |               |  Gaudi     |                 |  Gaudi     |
  |  Node 1    |               |  Node 2    |      ...        |  Node N    |
  |  8x Gaudi  |               |  8x Gaudi  |                 |  8x Gaudi  |
  |  cards     |               |  cards     |                 |  cards     |
  +-----+-----+               +-----+-----+                 +-----+-----+
        |                            |                             |
        +----------------------------+----------------------------+
                                     |
                     RoCEv2 Fabric (onboard NICs; direct
                     mesh for small clusters, leaf-spine
                     Ethernet switches for scale-out)
                                     |
                        +------------v------------+
                        |     Shared Storage        |
                        |   (Lustre / NFS / WEKA)   |
                        +----------------------------+
</code>

===== Networking Topology =====

  * **Gaudi 2:** 24 RoCEv2 ports per card, 21 used for scale-out (node-to-node) and 3 for scale-up (intra-node all-to-all mesh between the 8 accelerators).
  * **Gaudi 3:** same port layout, doubled per-port bandwidth (roughly 2x Gaudi 2 fabric throughput).
  * **Small clusters** (typically up to 32 nodes / 256 cards) can often wire nodes directly card-to-card in a mesh without external switches, cutting hardware cost.
  * **Larger clusters** need a leaf-spine Ethernet fabric with RoCEv2-capable switches (e.g. Arista, NVIDIA Spectrum, or similar 400GbE-class gear). Standard Ethernet, not a proprietary interconnect; this is the main operational difference from an InfiniBand-based NVIDIA fabric.
  * Lossless Ethernet configuration (PFC/ECN) on the switches matters for RoCEv2 performance; treat this the same way you'd tune an NVIDIA RoCE fabric, since there's no InfiniBand subnet manager to hide congestion behavior.

===== Host & Storage Considerations =====

  * **Host CPU/memory:** sized like our NVIDIA hosts, plenty of cores and RAM to keep the 8 accelerators fed; no Gaudi-specific host requirement beyond driver/firmware compatibility (see [[gaudi-software-firmware|Software & Firmware]]).
  * **Local NVMe:** recommended per node for checkpoint/scratch I/O, same as NVIDIA nodes.
  * **Shared storage:** reuse the existing parallel filesystem (Lustre/WEKA/etc.), no Gaudi-specific storage stack required.

===== Job Scheduling Notes =====

  * Tag Gaudi nodes with a distinct partition/label (Slurm partition or Kubernetes node label/taint) so standard NVIDIA jobs don't accidentally land on them and vice versa.
  * HCCL (Habana Collective Communications Library) is the NCCL equivalent; set `HCCL_*` env vars analogous to how `NCCL_*` vars are tuned today, particularly around scale-out topology hints if nodes aren't in a simple mesh.
  * Health checks: monitor for downed RoCE links and card resets the same way GPU health checks are wired into scheduler drain logic today.

----

[[gaudi-discontinued-clarification|← Previous]] | [[gaudi-guide|Guide Index]] | [[gaudi-software-firmware|Next →]]