Search stack operator notes

Live stack path on PD:

Live SearXNG config path on PD:

Known update-sensitive points:

Firecrawl containers share Gluetun's network namespace.
- firecrawl-playwright-service and firecrawl-api run with network_mode: service:search-gluetun.
- If search-gluetun is recreated during an update, those containers must be recreated too or they can die with namespace errors like:
  - joining network namespace of container ... No such container
  - getaddrinfo EAI_AGAIN firecrawl-redis
- The compose file now uses long-form depends_on with restart: true so explicit Compose updates to search-gluetun also restart the Firecrawl dependents.
SearXNG upstream can ship config drift.
- After an image update, compare /mnt/tank/docker/appdata/searxng/settings.yml against /mnt/tank/docker/appdata/searxng/settings.yml.new.
- Merge upstream changes deliberately instead of overwriting local behavior blindly.
- One confirmed breakage was an obsolete ask engine stanza lingering in the live settings after upstream removed that engine module; the fix was to remove the stale stanza and restart searxng.

Basic verification after updates:

sudo -n docker compose -f /mnt/docker-ssd/docker/compose/search/docker-compose.yaml ps
curl -sS "http://127.0.0.1:8888/search?q=test&format=json" | head
curl -sS http://127.0.0.1:3302/
optional functional scrape smoke test:
- POST {"url":"https://example.com"} to http://127.0.0.1:3302/v1/scrape

Firecrawl stability note on PD:

The upstream harness.js --start-docker launcher fans out into API + queue worker + extract worker + NUQ workers.
On PD, the default fan-out has caused first-start flaps (Port 3002 did not become available within 60000ms) and steady WORKER STALLED / Can't accept connection due to RAM/CPU load messages.
Keep the PD compose defaults conservative unless the host is re-sized: NUM_WORKERS_PER_QUEUE=1, CRAWL_CONCURRENT_REQUESTS=1, MAX_CONCURRENT_JOBS=1, BROWSER_POOL_SIZE=1, NUQ_WORKER_COUNT=1, and HARNESS_STARTUP_TIMEOUT_MS=180000.
After changing those values, force-recreate firecrawl-api and confirm curl -sS http://127.0.0.1:3302/ answers on the first start without an immediate harness timeout.

Rollback-first notes:

Back up docker-compose.yaml before live edits under /home/truenas_admin/doris-backups/
Back up settings.yml before SearXNG config edits under /home/truenas_admin/doris-backups/
If a Compose-side update breaks the stack and quick repair fails, restore the previous compose/config files and docker compose up -d the search stack again.

AI maintenance flow:

Homelab Docker maintenance is seeded into the Hermes homelab kanban board on a daily cadence.
The maintenance lane should treat this stack as update-sensitive and must run the verification commands above after any search-stack image change.
If the stack cannot be repaired quickly after an update, roll back first, then leave the exact failure evidence in the kanban task so alerts route cleanly.