Monitoring and Diagnostics

Monitoring and Diagnostics

This page covers how we watch Gaudi clusters day to day: what collects the data, what it measures, how alerts reach us, and where to look when something needs investigating.

The Monitoring Stack, Explained

We use the same monitoring stack for Gaudi as we do for our NVIDIA fleet. The only Gaudi-specific piece is the data collector at the start of the chain.

   Gaudi accelerators
          |
   habana-exporter          (reads hardware stats off each Gaudi card)
          |
     Prometheus              (stores the metrics over time)
          |
   Grafana dashboards        (visualizes them for humans)
          |
     Alertmanager            (decides who gets notified, and how)
          |
   PagerDuty / Slack         (where the actual alert lands)

What each piece does, in plain terms:

habana-exporter runs on every Gaudi node and reads live stats straight off the hardware: temperature, power draw, memory errors, network port health. It exposes these as an endpoint that Prometheus can read.
Prometheus polls that endpoint on a regular interval and stores the history, so we can see trends, not just a snapshot.
Grafana turns that stored history into dashboards, so a human can glance at a graph instead of reading raw numbers.
Alertmanager watches the same data and, when a threshold is crossed, decides which channel the alert should go to based on severity.
PagerDuty / Slack is where the alert actually shows up: PagerDuty for anything that needs someone paged immediately, Slack for anything lower-priority that just needs visibility.

Verifying the Exporter Is Running

If a node stops reporting metrics, this is the first thing to check.

# Start the exporter manually if it isn't already running
habana-exporter --port 9000 &
 
# Confirm it's serving metrics
curl http://localhost:9000/metrics | grep habana

If the curl command returns nothing, the exporter itself is the problem, not the accelerator. That's a different fix (restart the exporter service) than a genuine hardware fault.

Key Metrics We Track

These are the specific Prometheus metrics habana-exporter exposes, and the thresholds that trigger an alert:

Metric	What it measures	Alert threshold
`habana_aip_temperature`	Accelerator die temperature (°C)	Warn above 85°C, critical above 95°C
`habana_memory_used`	HBM memory used (bytes)	Warn above 90% of capacity
`habana_aip_power_draw`	Power draw (Watts)	Warn above 95% of rated TDP
`habana_aip_ecc_errors_correctable`	Correctable memory errors	Warn above 200 per day
`habana_aip_ecc_errors_uncorrectable`	Uncorrectable memory errors	Any at all triggers an alert (card should be retired)
`habana_aip_nic_rx_errors`	Network receive errors on the onboard RoCEv2 ports	Any at all triggers an alert
`habana_aip_nic_tx_errors`	Network transmit errors on the onboard RoCEv2 ports	Any at all triggers an alert

These map directly to the health baselines covered on Everyday Health Checks; this page is about how the data gets collected and surfaced, that page is about what “normal vs. urgent” looks like operationally.

Response Times When Something Fires an Alert

Severity	We acknowledge within	We respond within
Critical	15 minutes	30 minutes
Warning	1 hour	2 hours
Info	Next business day	N/A

Target: the cluster stays available at least 99.5% of the time, measured monthly.

How severity is decided: Alertmanager routes based on the metric and threshold that fired, not a manual judgment call. Anything tied to card health (temperature, ECC errors, power) or network link loss is Critical or Warning by default; slower-moving things like capacity trending toward a limit are typically Info.

Dashboards Available

Gaudi Cluster Overview: a heatmap of every node's temperature, side by side with HBM usage and RoCEv2 NIC error counts, so a degrading node is visible at a glance rather than buried in a table.
Node Health: a per-node drill-down view, useful once the overview dashboard has flagged something specific.
Fabric Health: RoCEv2 link status and error counters across the whole cluster, useful when a training job is running slower than expected and the cause might be the network rather than the accelerators.
Customer Report: the rolled-up monthly view we hand to the customer, combining availability, incident summary, and capacity trend.

Where to Look When Something Goes Wrong

When a node needs deeper investigation than the dashboards give you, these are the actual log locations and commands:

Source	Location / Command	What it tells you
Driver/kernel logs	`/var/log/habanalabs/`	Driver-level errors, firmware load failures
Kernel ring buffer	`dmesg \\| grep -i habana`	Low-level hardware events the kernel logged
systemd service logs	`journalctl -u habanalabs*`	Whether the driver service itself started/stopped cleanly
Extended diagnostics	`habana-validate` output	A full self-test of the card and its fabric connectivity
Container-level metrics	`habana-container-toolkit` Prometheus endpoint	Per-container accelerator usage, useful in Kubernetes environments

Important: pull and archive these logs *before* draining or replacing a node. They're often the only evidence Intel will accept when we file an RMA claim, so capturing them after the node is already swapped out is too late.

Everyday Health Checks — the routine schedule and threshold tables these metrics feed into.
Security — what happens once an issue is confirmed to be a security concern rather than a hardware one.
Hardware Replacement (RMA) — the next step once diagnostics confirm a card is actually bad.

← Previous | Guide Index | Next →

Table of Contents