This is an old revision of the document!
We run the same monitoring stack as our NVIDIA fleet, with a Gaudi-specific data collector:
habana-exporter → Prometheus → Grafana dashboards + Alertmanager → PagerDuty/Slack alerts.
| Severity | We acknowledge within | We respond within |
|---|---|---|
| Critical | 15 minutes | 30 minutes |
| Warning | 1 hour | 2 hours |
| Info | Next business day | — |
Target: cluster stays available 99.5%+ of the time, measured monthly.
| Source | Location / Command |
|---|---|
| Driver/kernel logs | /var/log/habanalabs/ |
| Kernel ring buffer | dmesg \| grep -i habana |
| systemd service logs | journalctl -u habanalabs* |
| Extended diagnostics | habana-validate output |
| Container-level metrics | habana-container-toolkit Prometheus endpoint |
We pull and archive these logs before draining or replacing a node — they're often the only evidence Intel will accept for an RMA claim.
