Differences

This shows you the differences between two versions of the page.

--- wiki:ai:gaudi-cluster-setup [2026/07/21 17:31] – created swilson
+++ wiki:ai:gaudi-cluster-setup [2026/07/21 17:45] (current) – swilson
@@ Line 4: / Line 4: @@
   * **Compute nodes:** bare-metal servers, 8 Gaudi accelerators each.
-  * **Node-to-node networking:** Gaudi chips have built-in RDMA networking ports (RoCEv2) — no separate InfiniBand card needed like on NVIDIA. Gaudi 3 can often connect node-to-node directly for small clusters; bigger clusters need external switches.
+  * **Node-to-node networking:** Gaudi chips have built-in RDMA networking ports (RoCEv2), no separate InfiniBand card needed like on NVIDIA. Gaudi 3 can often connect node-to-node directly for small clusters; bigger clusters need external switches.
-  * **Management network, storage, monitoring stack:** identical to our NVIDIA setup — same tools, same team.
+  * **Management network, storage, monitoring stack:** identical to our NVIDIA setup, same tools, same team.
   * **Job scheduling:** Slurm or Kubernetes, with Gaudi nodes kept in their own pool so jobs land on the right hardware.
+===== Architecture Diagram =====
+<code>
+                        +-------------------------+
+                        |   Management Plane      |
+                        |  (BCM, Prometheus,       |
+                        |   Grafana, Alertmanager) |
+                        +------------+------------+
+                                     |
+/10GbE OOB Management Network
+                                     |
+        +----------------------------+----------------------------+
+        |                            |                             |
+  +-----v-----+               +-----v-----+                 +-----v-----+
+  |  Gaudi     |               |  Gaudi     |                 |  Gaudi     |
+  |  Node 1    |               |  Node 2    |      ...        |  Node N    |
+  |  8x Gaudi  |               |  8x Gaudi  |                 |  8x Gaudi  |
+  |  cards     |               |  cards     |                 |  cards     |
+  +-----+-----+               +-----+-----+                 +-----+-----+
+        |                            |                             |
+        +----------------------------+----------------------------+
+                                     |
+                     RoCEv2 Fabric (onboard NICs; direct
+                     mesh for small clusters, leaf-spine
+                     Ethernet switches for scale-out)
+                                     |
+                        +------------v------------+
+                        |     Shared Storage        |
+                        |   (Lustre / NFS / WEKA)   |
+                        +----------------------------+
+</code>
+===== Networking Topology =====
+  * **Gaudi 2:** 24 RoCEv2 ports per card, 21 used for scale-out (node-to-node) and 3 for scale-up (intra-node all-to-all mesh between the 8 accelerators).
+  * **Gaudi 3:** same port layout, doubled per-port bandwidth (roughly 2x Gaudi 2 fabric throughput).
+  * **Small clusters** (typically up to 32 nodes / 256 cards) can often wire nodes directly card-to-card in a mesh without external switches, cutting hardware cost.
+  * **Larger clusters** need a leaf-spine Ethernet fabric with RoCEv2-capable switches (e.g. Arista, NVIDIA Spectrum, or similar 400GbE-class gear). Standard Ethernet, not a proprietary interconnect; this is the main operational difference from an InfiniBand-based NVIDIA fabric.
+  * Lossless Ethernet configuration (PFC/ECN) on the switches matters for RoCEv2 performance; treat this the same way you'd tune an NVIDIA RoCE fabric, since there's no InfiniBand subnet manager to hide congestion behavior.
+===== Host & Storage Considerations =====
+  * **Host CPU/memory:** sized like our NVIDIA hosts, plenty of cores and RAM to keep the 8 accelerators fed; no Gaudi-specific host requirement beyond driver/firmware compatibility (see [[gaudi-software-firmware|Software & Firmware]]).
+  * **Local NVMe:** recommended per node for checkpoint/scratch I/O, same as NVIDIA nodes.
+  * **Shared storage:** reuse the existing parallel filesystem (Lustre/WEKA/etc.), no Gaudi-specific storage stack required.
+===== Job Scheduling Notes =====
+  * Tag Gaudi nodes with a distinct partition/label (Slurm partition or Kubernetes node label/taint) so standard NVIDIA jobs don't accidentally land on them and vice versa.
+  * HCCL (Habana Collective Communications Library) is the NCCL equivalent; set `HCCL_*` env vars analogous to how `NCCL_*` vars are tuned today, particularly around scale-out topology hints if nodes aren't in a simple mesh.
+  * Health checks: monitor for downed RoCE links and card resets the same way GPU health checks are wired into scheduler drain logic today.
 ----

Combined Cloud Managed Services

Site Tools

Differences

Page Tools