Sync network redesign repo state and cutover artifacts
This commit is contained in:
135
home/doris-dashboard/docs/server-batch-validation-checklist.md
Normal file
135
home/doris-dashboard/docs/server-batch-validation-checklist.md
Normal file
@@ -0,0 +1,135 @@
|
||||
# Server Batch Validation Checklist
|
||||
|
||||
Use this immediately after moving Nomad, Serenity, and PD.
|
||||
Do not start Google nonsense, SSID cleanup, or extra cleanup until this is green.
|
||||
|
||||
Primary control path:
|
||||
- operator box: Rocinante
|
||||
- critical dependency group: Nomad + Serenity + PlausibleDeniability
|
||||
|
||||
---
|
||||
|
||||
## Before the batch
|
||||
|
||||
- [ ] Rocinante is stable and currently has UniFi/admin access.
|
||||
- [ ] Fallback admin path exists if Rocinante drops off.
|
||||
- [ ] Current port/profile screenshots taken for:
|
||||
- [ ] Port 22 -> Nomad
|
||||
- [ ] Port 24 -> Serenity
|
||||
- [ ] Port 23 -> PD
|
||||
- [ ] Expected target subnet/gateway for Servers VLAN is written down.
|
||||
- [ ] You are ready to unwind in reverse order if needed.
|
||||
|
||||
Move order:
|
||||
1. Nomad
|
||||
2. Serenity
|
||||
3. PD
|
||||
|
||||
Rollback order if ugly:
|
||||
1. PD
|
||||
2. Serenity
|
||||
3. Nomad
|
||||
|
||||
---
|
||||
|
||||
## Per-host quick checks during the wave
|
||||
|
||||
### After Nomad move
|
||||
- [ ] Link came back
|
||||
- [ ] Nomad got expected Servers IP/subnet/gateway
|
||||
- [ ] Rocinante can still reach Nomad
|
||||
- [ ] Expected Nomad-hosted service(s) respond
|
||||
- [ ] No obvious share/dependency explosion yet
|
||||
|
||||
### After Serenity move
|
||||
- [ ] Link came back
|
||||
- [ ] Serenity got expected Servers IP/subnet/gateway
|
||||
- [ ] Rocinante can still reach Serenity
|
||||
- [ ] Expected Serenity-hosted service(s) respond
|
||||
- [ ] No obvious share/dependency explosion yet
|
||||
|
||||
### After PD move
|
||||
- [ ] Link came back
|
||||
- [ ] PD got expected Servers IP/subnet/gateway
|
||||
- [ ] Rocinante can still reach PD
|
||||
- [ ] SSH/admin path to PD still works
|
||||
- [ ] Critical PD-hosted homelab services respond
|
||||
|
||||
---
|
||||
|
||||
## Immediate batch validation
|
||||
|
||||
Do all of these before touching anything else:
|
||||
|
||||
### Reachability
|
||||
- [ ] Rocinante -> Nomad reachable
|
||||
- [ ] Rocinante -> Serenity reachable
|
||||
- [ ] Rocinante -> PD reachable
|
||||
|
||||
### IP sanity
|
||||
- [ ] Nomad is on the intended Servers subnet
|
||||
- [ ] Serenity is on the intended Servers subnet
|
||||
- [ ] PD is on the intended Servers subnet
|
||||
- [ ] Default gateway is correct for all three
|
||||
|
||||
### Cross-host dependency sanity
|
||||
- [ ] NFS mounts/shares recovered cleanly
|
||||
- [ ] No host is hanging on stale routes or dead mount targets
|
||||
- [ ] Any name-resolution dependency still works as expected
|
||||
- [ ] No authentication/admin path you need for the rest of the window disappeared
|
||||
|
||||
### Service sanity
|
||||
- [ ] Doris/dashboard path(s) you need are still reachable
|
||||
- [ ] Any storage-backed services needed for the live window still work
|
||||
- [ ] Nothing critical is stuck waiting on a dead share
|
||||
|
||||
### Control-plane sanity
|
||||
- [ ] UniFi is still reachable from Rocinante
|
||||
- [ ] Switch/AP visibility still looks normal
|
||||
- [ ] No management-plane surprise coincided with the server batch
|
||||
|
||||
---
|
||||
|
||||
## Stop conditions
|
||||
|
||||
Stop and unwind if any of these are true:
|
||||
- [ ] PD is not reachable from Rocinante
|
||||
- [ ] NFS/share recovery is unclear after a short wait
|
||||
- [ ] You cannot tell whether the problem is routing, DNS, or service binding
|
||||
- [ ] UniFi/admin access is degraded at the same time
|
||||
- [ ] You are guessing instead of validating
|
||||
|
||||
---
|
||||
|
||||
## Reverse-order rollback strip
|
||||
|
||||
If the batch is bad and the reason is not immediately obvious:
|
||||
|
||||
1. Revert PD port/profile first.
|
||||
- [ ] PD returns to prior lane/profile
|
||||
- [ ] PD admin path restored
|
||||
2. Revert Serenity port/profile second.
|
||||
- [ ] Serenity returns to prior lane/profile
|
||||
3. Revert Nomad port/profile third.
|
||||
- [ ] Nomad returns to prior lane/profile
|
||||
4. Re-check:
|
||||
- [ ] Rocinante -> PD
|
||||
- [ ] Rocinante -> Serenity
|
||||
- [ ] Rocinante -> Nomad
|
||||
- [ ] UniFi/admin access stable again
|
||||
|
||||
Do not continue the broader migration until this set is stable.
|
||||
|
||||
---
|
||||
|
||||
## When this batch counts as successful
|
||||
|
||||
- [ ] Nomad moved successfully
|
||||
- [ ] Serenity moved successfully
|
||||
- [ ] PD moved successfully
|
||||
- [ ] Rocinante still has the admin path you need
|
||||
- [ ] Shares/services recovered cleanly enough to continue
|
||||
- [ ] No emergency broad firewall workaround was needed
|
||||
|
||||
If all six are true, move on.
|
||||
If not, stop pretending and fix or roll back first.
|
||||
Reference in New Issue
Block a user