Differences

This shows you the differences between two versions of the page.

--- wiki:ai:gaudi-monitoring-diagnostics [2026/07/21 17:32] – created swilson
+++ wiki:ai:gaudi-monitoring-diagnostics [2026/07/21 17:47] (current) – swilson
@@ Line 1: / Line 1: @@
 ====== Monitoring and Diagnostics ======
-We run the same monitoring stack as our NVIDIA fleet, with a Gaudi-specific data collector:
+This page covers how we watch Gaudi clusters day to day: what collects the data, what it measures, how alerts reach us, and where to look when something needs investigating.
-''habana-exporter'' → Prometheus → Grafana dashboards + Alertmanager → PagerDuty/Slack alerts.
+===== The Monitoring Stack, Explained =====
-==== Response times when something fires an alert ====
+We use the same monitoring stack for Gaudi as we do for our NVIDIA fleet. The only Gaudi-specific piece is the data collector at the start of the chain.
+<code>
+   Gaudi accelerators
+          |
+   habana-exporter          (reads hardware stats off each Gaudi card)
+          |
+     Prometheus              (stores the metrics over time)
+          |
+   Grafana dashboards        (visualizes them for humans)
+          |
+     Alertmanager            (decides who gets notified, and how)
+          |
+   PagerDuty / Slack         (where the actual alert lands)
+</code>
+**What each piece does, in plain terms:**
+  * **habana-exporter** runs on every Gaudi node and reads live stats straight off the hardware: temperature, power draw, memory errors, network port health. It exposes these as an endpoint that Prometheus can read.
+  * **Prometheus** polls that endpoint on a regular interval and stores the history, so we can see trends, not just a snapshot.
+  * **Grafana** turns that stored history into dashboards, so a human can glance at a graph instead of reading raw numbers.
+  * **Alertmanager** watches the same data and, when a threshold is crossed, decides which channel the alert should go to based on severity.
+  * **PagerDuty / Slack** is where the alert actually shows up: PagerDuty for anything that needs someone paged immediately, Slack for anything lower-priority that just needs visibility.
+===== Verifying the Exporter Is Running =====
+If a node stops reporting metrics, this is the first thing to check.
+<code bash>
+# Start the exporter manually if it isn't already running
+habana-exporter --port 9000 &
+# Confirm it's serving metrics
+curl http://localhost:9000/metrics | grep habana
+</code>
+If the ''curl'' command returns nothing, the exporter itself is the problem, not the accelerator. That's a different fix (restart the exporter service) than a genuine hardware fault.
+===== Key Metrics We Track =====
+These are the specific Prometheus metrics ''habana-exporter'' exposes, and the thresholds that trigger an alert:
+^ Metric                                  ^ What it measures                  ^ Alert threshold           ^
+| ''habana_aip_temperature''              | Accelerator die temperature (°C)  | Warn above 85°C, critical above 95°C |
+| ''habana_memory_used''                  | HBM memory used (bytes)           | Warn above 90% of capacity |
+| ''habana_aip_power_draw''                | Power draw (Watts)                | Warn above 95% of rated TDP |
+| ''habana_aip_ecc_errors_correctable''    | Correctable memory errors         | Warn above 200 per day    |
+| ''habana_aip_ecc_errors_uncorrectable''  | Uncorrectable memory errors       | Any at all triggers an alert (card should be retired) |
+| ''habana_aip_nic_rx_errors''             | Network receive errors on the onboard RoCEv2 ports | Any at all triggers an alert |
+| ''habana_aip_nic_tx_errors''             | Network transmit errors on the onboard RoCEv2 ports | Any at all triggers an alert |
+These map directly to the health baselines covered on [[gaudi-health-checks|Everyday Health Checks]]; this page is about how the data gets collected and surfaced, that page is about what "normal vs. urgent" looks like operationally.
+===== Response Times When Something Fires an Alert =====
 ^ Severity  ^ We acknowledge within  ^ We respond within  ^
 | Critical  | 15 minutes             | 30 minutes          |
 | Warning   | 1 hour                 | 2 hours             |
-| Info      | Next business day      | —                   |
+| Info      | Next business day      | N/A                 |
+**Target:** the cluster stays available at least 99.5% of the time, measured monthly.
+**How severity is decided:** Alertmanager routes based on the metric and threshold that fired, not a manual judgment call. Anything tied to card health (temperature, ECC errors, power) or network link loss is Critical or Warning by default; slower-moving things like capacity trending toward a limit are typically Info.
+===== Dashboards Available =====
+  * **Gaudi Cluster Overview:** a heatmap of every node's temperature, side by side with HBM usage and RoCEv2 NIC error counts, so a degrading node is visible at a glance rather than buried in a table.
+  * **Node Health:** a per-node drill-down view, useful once the overview dashboard has flagged something specific.
+  * **Fabric Health:** RoCEv2 link status and error counters across the whole cluster, useful when a training job is running slower than expected and the cause might be the network rather than the accelerators.
+  * **Customer Report:** the rolled-up monthly view we hand to the customer, combining availability, incident summary, and capacity trend.
+===== Where to Look When Something Goes Wrong =====
+When a node needs deeper investigation than the dashboards give you, these are the actual log locations and commands:
-**Target:** cluster stays available 99.5%+ of the time, measured monthly.
+^ Source                   ^ Location / Command                                    ^ What it tells you ^
+| Driver/kernel logs       | ''/var/log/habanalabs/''                              | Driver-level errors, firmware load failures |
+| Kernel ring buffer       | ''dmesg \| grep -i habana''                           | Low-level hardware events the kernel logged |
+| systemd service logs     | ''journalctl -u habanalabs*''                         | Whether the driver service itself started/stopped cleanly |
+| Extended diagnostics     | ''habana-validate'' output                            | A full self-test of the card and its fabric connectivity |
+| Container-level metrics  | ''habana-container-toolkit'' Prometheus endpoint      | Per-container accelerator usage, useful in Kubernetes environments |
-==== Where to look when something goes wrong ====
+**Important:** pull and archive these logs *before* draining or replacing a node. They're often the only evidence Intel will accept when we file an RMA claim, so capturing them after the node is already swapped out is too late.
-^ Source                   ^ Location / Command                                    ^
+===== Related Pages =====
-| Driver/kernel logs       | ''/var/log/habanalabs/''                              |
-| Kernel ring buffer       | ''dmesg \| grep -i habana''                           |
-| systemd service logs     | ''journalctl -u habanalabs*''                         |
-| Extended diagnostics      | ''habana-validate'' output                            |
-| Container-level metrics  | ''habana-container-toolkit'' Prometheus endpoint      |
-We pull and archive these logs **before** draining or replacing a node — they're often the only evidence Intel will accept for an RMA claim.
+  * [[gaudi-health-checks|Everyday Health Checks]] — the routine schedule and threshold tables these metrics feed into.
+  * [[gaudi-security|Security]] — what happens once an issue is confirmed to be a security concern rather than a hardware one.
+  * [[gaudi-rma|Hardware Replacement (RMA)]] — the next step once diagnostics confirm a card is actually bad.
 ----

Combined Cloud Managed Services

Site Tools

Differences

Page Tools