====== Storage and NVMe Health ======

Gaudi nodes have no accelerator-specific storage behavior, so this follows the standard fleet-wide process:

==== Local NVMe (scratch/cache) ====

  * **Weekly:** ''smartctl'' SMART health check on each node's local NVMe scratch/cache drive.
  * **Alert thresholds:** any media errors, drive life used over 80%, or more than 10 unsafe shutdowns in a month.
  * **Replacement:** a drive crossing the 80%-life-used mark is scheduled for replacement at the next maintenance window, drained, hot-swapped where the chassis supports it, filesystem-checked, then returned to service.
  * **Capacity monitoring:** local scratch usage is tracked alongside compute utilization; a node consistently running above 90% scratch capacity gets flagged so the customer can clean up stale checkpoints/datasets before it becomes a job-failure risk.
  * **Wear leveling:** since training/inference workloads are write-heavy on scratch, we track total bytes written (TBW) against the drive's rated endurance, not just the "life used" percentage SMART reports; this catches drives that are wearing out faster than the vendor's estimate assumed.

==== Shared storage (Lustre/NFS) ====

Where a cluster uses shared storage (checkpoints, datasets, shared home directories) rather than purely local scratch:

  * **Daily:** mount health check on every node; confirm the shared filesystem is mounted, writable, and responding within a normal latency window.
  * **Weekly:** capacity and inode usage trend review; free space and IOPS/throughput benchmark against baseline.
  * **Alert thresholds:** capacity above 85% (plan expansion), above 95% (urgent, jobs will start failing), or mount unavailable on any node (treated as a P2 incident since it can silently stall distributed training).
  * **Metadata server health (Lustre):** MDS/MDT health and failover status checked as part of the same daily sweep; a degraded MDS often shows up as slow ''ls''/directory operations before anything else breaks.
  * **Ownership:** the shared storage appliance/service itself may be customer-owned or a separate managed service depending on the SOW; Sirius's scope here is monitoring, alerting, and correlating storage issues with job failures, not necessarily storage hardware RMA.

==== Data protection expectations ====

  * Local NVMe scratch is **not backed up**; it's ephemeral by design, and anything that must survive a node reprovision needs to live on shared storage or be checkpointed off-node by the customer's training pipeline.
  * We do not manage backup of customer model checkpoints, datasets, or training outputs unless it's explicitly added to the SOW — see the capacity/asset and backup sections for what Sirius does own (management-plane config, not workload data).
  * When a drive is pulled (RMA, retirement, decommission), it goes through the same sanitization process as any other Gaudi hardware leaving the environment — see [[gaudi-rma|Hardware Replacement (RMA)]].

----

[[gaudi-os-patching-hardening|← Previous]] | [[gaudi-guide|Guide Index]] | [[gaudi-health-checks|Next →]]