User Tools

Site Tools


wiki:ai:gaudi-multi-tenancy
Draft | Approver: @ai-us-principals

Running Shared Clusters (Multi-Tenancy)

Many Gaudi clusters host more than one team or customer workload side by side. To keep tenants from interfering with each other:

  • Kubernetes: each tenant gets its own namespace with resource quotas capping how many Gaudi accelerators it can claim, plus network policies blocking cross-tenant traffic.
  • Slurm: each tenant gets its own account with a quality-of-service (QOS) policy limiting their share of the cluster.
  • Network fairness: on shared fabric, we apply traffic-priority rules so one tenant's heavy job can't starve another's network traffic.
  • Daily/weekly checks: confirm no nodes are stuck in a drained/cordoned state unexpectedly, and that the scheduler's control plane is healthy.

← Previous | Guide Index | Next →

wiki/ai/gaudi-multi-tenancy.txt · Last modified: by swilson