====== Handling Incidents and Making Changes ======

==== When something breaks, we classify it and respond on a clock ====

^ Priority       ^ Example                                   ^ Acknowledge          ^ Fix by             ^
| P1 — Critical  | Whole cluster down                        | 15 min                | 4 hr                |
| P2 — High      | Multiple nodes down, scheduler impacted   | 30 min                | 8 hr                |
| P3 — Medium    | Single node down                          | 2 hr                  | 24 hr               |
| P4 — Low       | Cosmetic/no active impact                 | Next business day     | 5 business days     |

Any P1/P2 incident gets a **written post-incident review** within 5 business days — what happened, why, and what we're changing to prevent a repeat.

==== When we need to change something ====

The level of process scales with the risk:

  * **Standard** — routine, pre-approved, no advance notice needed (e.g., applying an already-tested security patch).
  * **Normal** — needs 5 business days' notice and sign-off from our change review board.
  * **Significant** — needs 10 business days' notice //and// the customer's explicit sign-off (e.g., a firmware upgrade across the whole fleet).
  * **Emergency** — done immediately to stop active harm, with notification sent out as it happens.

Every change follows the same rollout pattern: test in a lab → soak in staging for 48 hours → get the change approved → roll out gradually (one node, then one rack, then the rest) → abort immediately if error rates rise or validation fails at any step.

----

[[gaudi-multi-tenancy|← Previous]] | [[gaudi-guide|Guide Index]] | [[gaudi-capacity-planning|Next →]]