This page covers how we watch Gaudi clusters day to day: what collects the data, what it measures, how alerts reach us, and where to look when something needs investigating.

We use the same monitoring stack for Gaudi as we do for our NVIDIA fleet. The only Gaudi-specific piece is the data collector at the start of the chain.

   Gaudi accelerators
          |
   habana-exporter          (reads hardware stats off each Gaudi card)
          |
     Prometheus              (stores the metrics over time)
          |
   Grafana dashboards        (visualizes them for humans)
          |
     Alertmanager            (decides who gets notified, and how)
          |
   PagerDuty / Slack         (where the actual alert lands)

What each piece does, in plain terms:

• habana-exporter runs on every Gaudi node and reads live stats straight off the hardware: temperature, power draw, memory errors, network port health. It exposes these as an endpoint that Prometheus can read.
• Prometheus polls that endpoint on a regular interval and stores the history, so we can see trends, not just a snapshot.
• Grafana turns that stored history into dashboards, so a human can glance at a graph instead of reading raw numbers.
• Alertmanager watches the same data and, when a threshold is crossed, decides which channel the alert should go to based on severity.
• PagerDuty / Slack is where the alert actually shows up: PagerDuty for anything that needs someone paged immediately, Slack for anything lower-priority that just needs visibility.

If a node stops reporting metrics, this is the first thing to check.

# Start the exporter manually if it isn't already running
habana-exporter --port 9000 &
 
# Confirm it's serving metrics
curl http://localhost:9000/metrics | grep habana

If the curl command returns nothing, the exporter itself is the problem, not the accelerator. That's a different fix (restart the exporter service) than a genuine hardware fault.

These are the specific Prometheus metrics habana-exporter exposes, and the thresholds that trigger an alert:

These map directly to the health baselines covered on Everyday Health Checks; this page is about how the data gets collected and surfaced, that page is about what “normal vs. urgent” looks like operationally.

Target: the cluster stays available at least 99.5% of the time, measured monthly.

How severity is decided: Alertmanager routes based on the metric and threshold that fired, not a manual judgment call. Anything tied to card health (temperature, ECC errors, power) or network link loss is Critical or Warning by default; slower-moving things like capacity trending toward a limit are typically Info.

• Gaudi Cluster Overview: a heatmap of every node's temperature, side by side with HBM usage and RoCEv2 NIC error counts, so a degrading node is visible at a glance rather than buried in a table.
• Node Health: a per-node drill-down view, useful once the overview dashboard has flagged something specific.
• Fabric Health: RoCEv2 link status and error counters across the whole cluster, useful when a training job is running slower than expected and the cause might be the network rather than the accelerators.
• Customer Report: the rolled-up monthly view we hand to the customer, combining availability, incident summary, and capacity trend.

When a node needs deeper investigation than the dashboards give you, these are the actual log locations and commands:

Important: pull and archive these logs *before* draining or replacing a node. They're often the only evidence Intel will accept when we file an RMA claim, so capturing them after the node is already swapped out is too late.

• Everyday Health Checks — the routine schedule and threshold tables these metrics feed into.
• Security — what happens once an issue is confirmed to be a security concern rather than a hardware one.
• Hardware Replacement (RMA) — the next step once diagnostics confirm a card is actually bad.

← Previous | Guide Index | Next →