# Search stack operator notes

Live stack path on PD:
- `/mnt/docker-ssd/docker/compose/search`

Live SearXNG config path on PD:
- `/mnt/tank/docker/appdata/searxng/settings.yml`

Known update-sensitive points:

1. Firecrawl containers share Gluetun's network namespace.
   - `firecrawl-playwright-service` and `firecrawl-api` run with `network_mode: service:search-gluetun`.
   - If `search-gluetun` is recreated during an update, those containers must be recreated too or they can die with namespace errors like:
     - `joining network namespace of container ... No such container`
     - `getaddrinfo EAI_AGAIN firecrawl-redis`
   - The compose file now uses long-form `depends_on` with `restart: true` so explicit Compose updates to `search-gluetun` also restart the Firecrawl dependents.

2. SearXNG upstream can ship config drift.
   - After an image update, compare `/mnt/tank/docker/appdata/searxng/settings.yml` against `/mnt/tank/docker/appdata/searxng/settings.yml.new`.
   - Merge upstream changes deliberately instead of overwriting local behavior blindly.
   - One confirmed breakage was an obsolete `ask` engine stanza lingering in the live settings after upstream removed that engine module; the fix was to remove the stale stanza and restart `searxng`.

Basic verification after updates:
- `sudo -n docker compose -f /mnt/docker-ssd/docker/compose/search/docker-compose.yaml ps`
- `curl -sS "http://127.0.0.1:8888/search?q=test&format=json" | head`
- `curl -sS http://127.0.0.1:3302/`
- optional functional scrape smoke test:
  - POST `{"url":"https://example.com"}` to `http://127.0.0.1:3302/v1/scrape`

Firecrawl stability note on PD:
- The upstream `harness.js --start-docker` launcher fans out into API + queue worker + extract worker + NUQ workers.
- On PD, the default fan-out has caused first-start flaps (`Port 3002 did not become available within 60000ms`) and steady `WORKER STALLED` / `Can't accept connection due to RAM/CPU load` messages.
- Keep the PD compose defaults conservative unless the host is re-sized: `NUM_WORKERS_PER_QUEUE=1`, `CRAWL_CONCURRENT_REQUESTS=1`, `MAX_CONCURRENT_JOBS=1`, `BROWSER_POOL_SIZE=1`, `NUQ_WORKER_COUNT=1`, and `HARNESS_STARTUP_TIMEOUT_MS=180000`.
- After changing those values, force-recreate `firecrawl-api` and confirm `curl -sS http://127.0.0.1:3302/` answers on the first start without an immediate harness timeout.

Rollback-first notes:
- Back up `docker-compose.yaml` before live edits under `/home/truenas_admin/doris-backups/`
- Back up `settings.yml` before SearXNG config edits under `/home/truenas_admin/doris-backups/`
- If a Compose-side update breaks the stack and quick repair fails, restore the previous compose/config files and `docker compose up -d` the search stack again.

AI maintenance flow:
- Homelab Docker maintenance is seeded into the Hermes `homelab` kanban board on a daily cadence.
- The maintenance lane should treat this stack as update-sensitive and must run the verification commands above after any search-stack image change.
- If the stack cannot be repaired quickly after an update, roll back first, then leave the exact failure evidence in the kanban task so alerts route cleanly.