Everyday Health Checks

How Often	What We Check
Daily (automated)	Every node's status via `hl-smi`, temperature, power draw, memory error counts.
Weekly	Driver health, network port errors, cross-check against any training failures.
Monthly	Firmware versions against baseline, full diagnostic sweep, scan for new security advisories.
Quarterly	Full report to the customer, spare-parts inventory check, migration timeline check-in.

When something's wrong, here's what "wrong" looks like

Signal	Normal	Worth a Look	Urgent
Chip temperature	30–80°C	above 85°C	above 95°C
Memory temperature	30–85°C	above 88°C	above 95°C
Power draw	under 95% of max	sustained above 95%	throttling kicks in
Uncorrectable memory errors	0	any at all	any at all — retire the card
Network link errors	0	any at all	more than 5/hour

We keep spare Gaudi cards on-site at all times — that's our main safety net, since Intel's replacement turnaround isn't as fast as it used to be for an actively-marketed product.

← Previous | Guide Index | Next →