Handling Incidents and Making Changes
- - When something breaks, we classify it and respond on a clock
  - When we need to change something

Handling Incidents and Making Changes

When something breaks, we classify it and respond on a clock

Priority	Example	Acknowledge	Fix by
P1 — Critical	Whole cluster down	15 min	4 hr
P2 — High	Multiple nodes down, scheduler impacted	30 min	8 hr
P3 — Medium	Single node down	2 hr	24 hr
P4 — Low	Cosmetic/no active impact	Next business day	5 business days

Any P1/P2 incident gets a written post-incident review within 5 business days — what happened, why, and what we're changing to prevent a repeat.

When we need to change something

The level of process scales with the risk:

Standard — routine, pre-approved, no advance notice needed (e.g., applying an already-tested security patch).
Normal — needs 5 business days' notice and sign-off from our change review board.
Significant — needs 10 business days' notice and the customer's explicit sign-off (e.g., a firmware upgrade across the whole fleet).
Emergency — done immediately to stop active harm, with notification sent out as it happens.

Every change follows the same rollout pattern: test in a lab → soak in staging for 48 hours → get the change approved → roll out gradually (one node, then one rack, then the rest) → abort immediately if error rates rise or validation fails at any step.

← Previous | Guide Index | Next →

Table of Contents

Handling Incidents and Making Changes

When something breaks, we classify it and respond on a clock

When we need to change something