truenas-stacks/home/doris-dashboard/docs/server-batch-validation-checklist.md

Server Batch Validation Checklist

Use this immediately after moving Nomad, Serenity, and PD. Do not start Google nonsense, SSID cleanup, or extra cleanup until this is green.

Primary control path:

• operator box: Rocinante
• critical dependency group: Nomad + Serenity + PlausibleDeniability

---

Before the batch

• Rocinante is stable and currently has UniFi/admin access.
• Fallback admin path exists if Rocinante drops off.
• Current port/profile screenshots taken for:
  • Port 22 -> Nomad
  • Port 24 -> Serenity
  • Port 23 -> PD
• Expected target subnet/gateway for Servers VLAN is written down.
• You are ready to unwind in reverse order if needed.

Move order:

1. Nomad
2. Serenity
3. PD

Rollback order if ugly:

1. PD
2. Serenity
3. Nomad

---

Per-host quick checks during the wave

After Nomad move

• Link came back
• Nomad got expected Servers IP/subnet/gateway
• Rocinante can still reach Nomad
• Expected Nomad-hosted service(s) respond
• No obvious share/dependency explosion yet

After Serenity move

• Link came back
• Serenity got expected Servers IP/subnet/gateway
• Rocinante can still reach Serenity
• Expected Serenity-hosted service(s) respond
• No obvious share/dependency explosion yet

After PD move

• Link came back
• PD got expected Servers IP/subnet/gateway
• Rocinante can still reach PD
• SSH/admin path to PD still works
• Critical PD-hosted homelab services respond

---

Immediate batch validation

Do all of these before touching anything else:

Reachability

• Rocinante -> Nomad reachable
• Rocinante -> Serenity reachable
• Rocinante -> PD reachable

IP sanity

• Nomad is on the intended Servers subnet
• Serenity is on the intended Servers subnet
• PD is on the intended Servers subnet
• Default gateway is correct for all three

Cross-host dependency sanity

• NFS mounts/shares recovered cleanly
• No host is hanging on stale routes or dead mount targets
• Any name-resolution dependency still works as expected
• No authentication/admin path you need for the rest of the window disappeared

Service sanity

• Doris/dashboard path(s) you need are still reachable
• Any storage-backed services needed for the live window still work
• Nothing critical is stuck waiting on a dead share

Control-plane sanity

• UniFi is still reachable from Rocinante
• Switch/AP visibility still looks normal
• No management-plane surprise coincided with the server batch

---

Stop conditions

Stop and unwind if any of these are true:

• PD is not reachable from Rocinante
• NFS/share recovery is unclear after a short wait
• You cannot tell whether the problem is routing, DNS, or service binding
• UniFi/admin access is degraded at the same time
• You are guessing instead of validating

---

Reverse-order rollback strip

If the batch is bad and the reason is not immediately obvious:

1. Revert PD port/profile first.
   • PD returns to prior lane/profile
   • PD admin path restored
2. Revert Serenity port/profile second.
   • Serenity returns to prior lane/profile
3. Revert Nomad port/profile third.
   • Nomad returns to prior lane/profile
4. Re-check:
   • Rocinante -> PD
   • Rocinante -> Serenity
   • Rocinante -> Nomad
   • UniFi/admin access stable again

Do not continue the broader migration until this set is stable.

---

When this batch counts as successful

• Nomad moved successfully
• Serenity moved successfully
• PD moved successfully
• Rocinante still has the admin path you need
• Shares/services recovered cleanly enough to continue
• No emergency broad firewall workaround was needed

If all six are true, move on. If not, stop pretending and fix or roll back first.