Storage and NVMe Health

Gaudi nodes have no accelerator-specific storage behavior, so this follows the standard fleet-wide process:

Weekly: smartctl SMART health check on each node's local NVMe scratch/cache drive.
Alert thresholds: any media errors, drive life used over 80%, or more than 10 unsafe shutdowns in a month.
Replacement: a drive crossing the 80%-life-used mark is scheduled for replacement at the next maintenance window, drained, hot-swapped where the chassis supports it, filesystem-checked, then returned to service.
Capacity monitoring: local scratch usage is tracked alongside compute utilization; a node consistently running above 90% scratch capacity gets flagged so the customer can clean up stale checkpoints/datasets before it becomes a job-failure risk.
Wear leveling: since training/inference workloads are write-heavy on scratch, we track total bytes written (TBW) against the drive's rated endurance, not just the “life used” percentage SMART reports; this catches drives that are wearing out faster than the vendor's estimate assumed.

Where a cluster uses shared storage (checkpoints, datasets, shared home directories) rather than purely local scratch:

Daily: mount health check on every node; confirm the shared filesystem is mounted, writable, and responding within a normal latency window.
Weekly: capacity and inode usage trend review; free space and IOPS/throughput benchmark against baseline.
Alert thresholds: capacity above 85% (plan expansion), above 95% (urgent, jobs will start failing), or mount unavailable on any node (treated as a P2 incident since it can silently stall distributed training).
Metadata server health (Lustre): MDS/MDT health and failover status checked as part of the same daily sweep; a degraded MDS often shows up as slow ls/directory operations before anything else breaks.
Ownership: the shared storage appliance/service itself may be customer-owned or a separate managed service depending on the SOW; Sirius's scope here is monitoring, alerting, and correlating storage issues with job failures, not necessarily storage hardware RMA.

Local NVMe scratch is not backed up; it's ephemeral by design, and anything that must survive a node reprovision needs to live on shared storage or be checkpointed off-node by the customer's training pipeline.
We do not manage backup of customer model checkpoints, datasets, or training outputs unless it's explicitly added to the SOW — see the capacity/asset and backup sections for what Sirius does own (management-plane config, not workload data).
When a drive is pulled (RMA, retirement, decommission), it goes through the same sanitization process as any other Gaudi hardware leaving the environment — see Hardware Replacement (RMA).