====== Monitoring and Diagnostics ======
This page covers how we watch Gaudi clusters day to day: what collects the data, what it measures, how alerts reach us, and where to look when something needs investigating.
===== The Monitoring Stack, Explained =====
We use the same monitoring stack for Gaudi as we do for our NVIDIA fleet. The only Gaudi-specific piece is the data collector at the start of the chain.
Gaudi accelerators
|
habana-exporter (reads hardware stats off each Gaudi card)
|
Prometheus (stores the metrics over time)
|
Grafana dashboards (visualizes them for humans)
|
Alertmanager (decides who gets notified, and how)
|
PagerDuty / Slack (where the actual alert lands)
**What each piece does, in plain terms:**
* **habana-exporter** runs on every Gaudi node and reads live stats straight off the hardware: temperature, power draw, memory errors, network port health. It exposes these as an endpoint that Prometheus can read.
* **Prometheus** polls that endpoint on a regular interval and stores the history, so we can see trends, not just a snapshot.
* **Grafana** turns that stored history into dashboards, so a human can glance at a graph instead of reading raw numbers.
* **Alertmanager** watches the same data and, when a threshold is crossed, decides which channel the alert should go to based on severity.
* **PagerDuty / Slack** is where the alert actually shows up: PagerDuty for anything that needs someone paged immediately, Slack for anything lower-priority that just needs visibility.
===== Verifying the Exporter Is Running =====
If a node stops reporting metrics, this is the first thing to check.
# Start the exporter manually if it isn't already running
habana-exporter --port 9000 &
# Confirm it's serving metrics
curl http://localhost:9000/metrics | grep habana
If the ''curl'' command returns nothing, the exporter itself is the problem, not the accelerator. That's a different fix (restart the exporter service) than a genuine hardware fault.
===== Key Metrics We Track =====
These are the specific Prometheus metrics ''habana-exporter'' exposes, and the thresholds that trigger an alert:
^ Metric ^ What it measures ^ Alert threshold ^
| ''habana_aip_temperature'' | Accelerator die temperature (°C) | Warn above 85°C, critical above 95°C |
| ''habana_memory_used'' | HBM memory used (bytes) | Warn above 90% of capacity |
| ''habana_aip_power_draw'' | Power draw (Watts) | Warn above 95% of rated TDP |
| ''habana_aip_ecc_errors_correctable'' | Correctable memory errors | Warn above 200 per day |
| ''habana_aip_ecc_errors_uncorrectable'' | Uncorrectable memory errors | Any at all triggers an alert (card should be retired) |
| ''habana_aip_nic_rx_errors'' | Network receive errors on the onboard RoCEv2 ports | Any at all triggers an alert |
| ''habana_aip_nic_tx_errors'' | Network transmit errors on the onboard RoCEv2 ports | Any at all triggers an alert |
These map directly to the health baselines covered on [[gaudi-health-checks|Everyday Health Checks]]; this page is about how the data gets collected and surfaced, that page is about what "normal vs. urgent" looks like operationally.
===== Response Times When Something Fires an Alert =====
^ Severity ^ We acknowledge within ^ We respond within ^
| Critical | 15 minutes | 30 minutes |
| Warning | 1 hour | 2 hours |
| Info | Next business day | N/A |
**Target:** the cluster stays available at least 99.5% of the time, measured monthly.
**How severity is decided:** Alertmanager routes based on the metric and threshold that fired, not a manual judgment call. Anything tied to card health (temperature, ECC errors, power) or network link loss is Critical or Warning by default; slower-moving things like capacity trending toward a limit are typically Info.
===== Dashboards Available =====
* **Gaudi Cluster Overview:** a heatmap of every node's temperature, side by side with HBM usage and RoCEv2 NIC error counts, so a degrading node is visible at a glance rather than buried in a table.
* **Node Health:** a per-node drill-down view, useful once the overview dashboard has flagged something specific.
* **Fabric Health:** RoCEv2 link status and error counters across the whole cluster, useful when a training job is running slower than expected and the cause might be the network rather than the accelerators.
* **Customer Report:** the rolled-up monthly view we hand to the customer, combining availability, incident summary, and capacity trend.
===== Where to Look When Something Goes Wrong =====
When a node needs deeper investigation than the dashboards give you, these are the actual log locations and commands:
^ Source ^ Location / Command ^ What it tells you ^
| Driver/kernel logs | ''/var/log/habanalabs/'' | Driver-level errors, firmware load failures |
| Kernel ring buffer | ''dmesg \| grep -i habana'' | Low-level hardware events the kernel logged |
| systemd service logs | ''journalctl -u habanalabs*'' | Whether the driver service itself started/stopped cleanly |
| Extended diagnostics | ''habana-validate'' output | A full self-test of the card and its fabric connectivity |
| Container-level metrics | ''habana-container-toolkit'' Prometheus endpoint | Per-container accelerator usage, useful in Kubernetes environments |
**Important:** pull and archive these logs *before* draining or replacing a node. They're often the only evidence Intel will accept when we file an RMA claim, so capturing them after the node is already swapped out is too late.
===== Related Pages =====
* [[gaudi-health-checks|Everyday Health Checks]] — the routine schedule and threshold tables these metrics feed into.
* [[gaudi-security|Security]] — what happens once an issue is confirmed to be a security concern rather than a hardware one.
* [[gaudi-rma|Hardware Replacement (RMA)]] — the next step once diagnostics confirm a card is actually bad.
----
[[gaudi-health-checks|← Previous]] | [[gaudi-guide|Guide Index]] | [[gaudi-security|Next →]]