Stabilize Firecrawl startup on PD
Some checks failed
secret-guardrails / artifact-secret-scan (push) Has been cancelled
secret-guardrails / gitleaks (push) Has been cancelled

This commit is contained in:
Fizzlepoof
2026-06-07 04:14:23 +00:00
parent 7a67a2febc
commit 28b2edf2a3
2 changed files with 12 additions and 5 deletions

View File

@@ -27,6 +27,12 @@ Basic verification after updates:
- optional functional scrape smoke test:
- POST `{"url":"https://example.com"}` to `http://127.0.0.1:3302/v1/scrape`
Firecrawl stability note on PD:
- The upstream `harness.js --start-docker` launcher fans out into API + queue worker + extract worker + NUQ workers.
- On PD, the default fan-out has caused first-start flaps (`Port 3002 did not become available within 60000ms`) and steady `WORKER STALLED` / `Can't accept connection due to RAM/CPU load` messages.
- Keep the PD compose defaults conservative unless the host is re-sized: `NUM_WORKERS_PER_QUEUE=1`, `CRAWL_CONCURRENT_REQUESTS=1`, `MAX_CONCURRENT_JOBS=1`, `BROWSER_POOL_SIZE=1`, `NUQ_WORKER_COUNT=1`, and `HARNESS_STARTUP_TIMEOUT_MS=180000`.
- After changing those values, force-recreate `firecrawl-api` and confirm `curl -sS http://127.0.0.1:3302/` answers on the first start without an immediate harness timeout.
Rollback-first notes:
- Back up `docker-compose.yaml` before live edits under `/home/truenas_admin/doris-backups/`
- Back up `settings.yml` before SearXNG config edits under `/home/truenas_admin/doris-backups/`