Many Gaudi clusters host more than one team or customer workload side by side, rather than being dedicated to a single customer. This page covers how we keep those tenants from interfering with each other, at both the scheduler level and the network level.

On a shared cluster, one team's misbehaving job can, if left unchecked, starve every other team of compute or network bandwidth. The goal of everything below is to make that structurally impossible rather than relying on tenants to behave well.

• Namespaces: each tenant is given its own Kubernetes namespace. This is the basic unit of separation; resources, secrets, and configuration in one tenant's namespace are not visible to another tenant by default.
• Resource quotas: a ResourceQuota object on each namespace caps how many Gaudi accelerators (via the habana.ai/gaudi resource) that tenant can claim at once. Without this, a single tenant could request every accelerator in the cluster and starve everyone else.
• LimitRange: sets sane defaults and maximums for individual pod requests, so one misconfigured job can't accidentally request an unreasonable amount of resource inside the quota.
• Network policies: NetworkPolicy objects block pod-to-pod traffic across namespaces by default. Cross-tenant traffic is only allowed if explicitly opened up, which we do not do unless the SOW calls for shared services between tenants.
• RBAC: access to each namespace is scoped through Kubernetes role-based access control, so a tenant's credentials only grant permissions inside their own namespace, not cluster-wide.

• Accounts: each tenant gets a distinct Slurm account, which is how usage and fair-share accounting is tracked per tenant.
• Quality of Service (QOS) policies: a QOS policy caps how much of the cluster (nodes, cores, GPU/accelerator count, wall-clock time) a given account can consume concurrently. This is the Slurm equivalent of a Kubernetes ResourceQuota.
• Partitions: Gaudi nodes are typically kept in their own partition, separate from any NVIDIA partition in a mixed fleet, so jobs land on the accelerator type they were actually written for.
• Fair-share scheduling: Slurm's fair-share algorithm de-prioritizes a tenant's future jobs if they've been consuming a disproportionate share of the cluster recently, which prevents one heavy user from permanently crowding out others even within their allowed quota.

Kubernetes namespaces and Slurm QOS policies control compute access, but on a cluster with a shared RoCEv2 fabric, one tenant's job can still flood the network and slow everyone else's distributed training down, even if their compute usage is within quota.

• Traffic-priority classes (QOS/PFC): we apply Priority Flow Control classes on the switch fabric so no single tenant's traffic can monopolize bandwidth and starve another tenant's collective communication (allreduce, etc.) traffic.
• Why this is separate from compute quotas: a tenant staying within their accelerator quota can still generate a disproportionate amount of network traffic per accelerator, depending on their model's communication pattern. Network fairness has to be enforced independently of compute fairness for this reason.

A node stuck in an unexpected drained or cordoned state is a common early symptom of a tenant-isolation problem: it usually means a job failed to release its resources cleanly, and left the node unavailable for the next tenant's workload.

If a tenant does manage to exceed its allotted share, either through a quota misconfiguration or, more rarely, an actual scheduler bug, this is handled as a standard incident (see Handling Incidents and Making Changes), not silently absorbed. The affected tenant's workload is not automatically throttled retroactively; we investigate root cause, fix the quota/policy gap, and document it, since a recurring quota bypass is a configuration-drift issue (see Keeping Configuration Consistent).

• Keeping Configuration Consistent — how we catch quota/policy drift before it becomes a real problem.
• Handling Incidents and Making Changes — what happens when isolation actually fails.
• Cluster Setup — where the RoCEv2 fabric this page's network fairness section depends on is described.

← Previous | Guide Index | Next →