Differences

This shows you the differences between two versions of the page.

--- wiki:ai:gaudi-storage-nvme [2026/07/21 17:32] – created swilson
+++ wiki:ai:gaudi-storage-nvme [2026/07/21 17:40] (current) – swilson
@@ Line 2: / Line 2: @@
 Gaudi nodes have no accelerator-specific storage behavior, so this follows the standard fleet-wide process:
+==== Local NVMe (scratch/cache) ====
   * **Weekly:** ''smartctl'' SMART health check on each node's local NVMe scratch/cache drive.
   * **Alert thresholds:** any media errors, drive life used over 80%, or more than 10 unsafe shutdowns in a month.
-  * **Replacement:** a drive crossing the 80%-life-used mark is scheduled for replacement at the next maintenance window — drained, hot-swapped where the chassis supports it, filesystem-checked, then returned to service.
+  * **Replacement:** a drive crossing the 80%-life-used mark is scheduled for replacement at the next maintenance window, drained, hot-swapped where the chassis supports it, filesystem-checked, then returned to service.
+  * **Capacity monitoring:** local scratch usage is tracked alongside compute utilization; a node consistently running above 90% scratch capacity gets flagged so the customer can clean up stale checkpoints/datasets before it becomes a job-failure risk.
+  * **Wear leveling:** since training/inference workloads are write-heavy on scratch, we track total bytes written (TBW) against the drive's rated endurance, not just the "life used" percentage SMART reports; this catches drives that are wearing out faster than the vendor's estimate assumed.
+==== Shared storage (Lustre/NFS) ====
+Where a cluster uses shared storage (checkpoints, datasets, shared home directories) rather than purely local scratch:
+  * **Daily:** mount health check on every node; confirm the shared filesystem is mounted, writable, and responding within a normal latency window.
+  * **Weekly:** capacity and inode usage trend review; free space and IOPS/throughput benchmark against baseline.
+  * **Alert thresholds:** capacity above 85% (plan expansion), above 95% (urgent, jobs will start failing), or mount unavailable on any node (treated as a P2 incident since it can silently stall distributed training).
+  * **Metadata server health (Lustre):** MDS/MDT health and failover status checked as part of the same daily sweep; a degraded MDS often shows up as slow ''ls''/directory operations before anything else breaks.
+  * **Ownership:** the shared storage appliance/service itself may be customer-owned or a separate managed service depending on the SOW; Sirius's scope here is monitoring, alerting, and correlating storage issues with job failures, not necessarily storage hardware RMA.
+==== Data protection expectations ====
+  * Local NVMe scratch is **not backed up**; it's ephemeral by design, and anything that must survive a node reprovision needs to live on shared storage or be checkpointed off-node by the customer's training pipeline.
+  * We do not manage backup of customer model checkpoints, datasets, or training outputs unless it's explicitly added to the SOW — see the capacity/asset and backup sections for what Sirius does own (management-plane config, not workload data).
+  * When a drive is pulled (RMA, retirement, decommission), it goes through the same sanitization process as any other Gaudi hardware leaving the environment — see [[gaudi-rma|Hardware Replacement (RMA)]].
 ----

Combined Cloud Managed Services

Site Tools

Differences

Page Tools