fizzlepoof/truenas-stacks
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Stabilize Firecrawl startup on PD
Some checks failed
secret-guardrails / artifact-secret-scan (push) Has been cancelled
Details
secret-guardrails / gitleaks (push) Has been cancelled
Details

Browse Source
This commit is contained in:
Fizzlepoof
2026-06-07 04:14:23 +00:00
parent 7a67a2febc
commit 28b2edf2a3
2 changed files with 12 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
search/README.md
View File

@@ -27,6 +27,12 @@ Basic verification after updates:
- optional functional scrape smoke test: - optional functional scrape smoke test:
- POST `{"url":"https://example.com"}` to `http://127.0.0.1:3302/v1/scrape` - POST `{"url":"https://example.com"}` to `http://127.0.0.1:3302/v1/scrape`
Firecrawl stability note on PD:
- The upstream `harness.js --start-docker` launcher fans out into API + queue worker + extract worker + NUQ workers.
- On PD, the default fan-out has caused first-start flaps (`Port 3002 did not become available within 60000ms`) and steady `WORKER STALLED` / `Can't accept connection due to RAM/CPU load` messages.
- Keep the PD compose defaults conservative unless the host is re-sized: `NUM_WORKERS_PER_QUEUE=1`, `CRAWL_CONCURRENT_REQUESTS=1`, `MAX_CONCURRENT_JOBS=1`, `BROWSER_POOL_SIZE=1`, `NUQ_WORKER_COUNT=1`, and `HARNESS_STARTUP_TIMEOUT_MS=180000`.
- After changing those values, force-recreate `firecrawl-api` and confirm `curl -sS http://127.0.0.1:3302/` answers on the first start without an immediate harness timeout.
Rollback-first notes: Rollback-first notes:
- Back up `docker-compose.yaml` before live edits under `/home/truenas_admin/doris-backups/` - Back up `docker-compose.yaml` before live edits under `/home/truenas_admin/doris-backups/`
- Back up `settings.yml` before SearXNG config edits under `/home/truenas_admin/doris-backups/` - Back up `settings.yml` before SearXNG config edits under `/home/truenas_admin/doris-backups/`

11
search/docker-compose.yaml
View File

@@ -126,10 +126,11 @@ services:
- POSTGRES_HOST=firecrawl-postgres - POSTGRES_HOST=firecrawl-postgres
- POSTGRES_PORT=5432 - POSTGRES_PORT=5432
- USE_DB_AUTHENTICATION=${USE_DB_AUTHENTICATION:-false} - USE_DB_AUTHENTICATION=${USE_DB_AUTHENTICATION:-false}
- NUM_WORKERS_PER_QUEUE=${NUM_WORKERS_PER_QUEUE:-4} - NUM_WORKERS_PER_QUEUE=${NUM_WORKERS_PER_QUEUE:-1}
- CRAWL_CONCURRENT_REQUESTS=${CRAWL_CONCURRENT_REQUESTS:-4} - CRAWL_CONCURRENT_REQUESTS=${CRAWL_CONCURRENT_REQUESTS:-1}
- MAX_CONCURRENT_JOBS=${MAX_CONCURRENT_JOBS:-2} - MAX_CONCURRENT_JOBS=${MAX_CONCURRENT_JOBS:-1}
- BROWSER_POOL_SIZE=${BROWSER_POOL_SIZE:-2} - BROWSER_POOL_SIZE=${BROWSER_POOL_SIZE:-1}
- NUQ_WORKER_COUNT=${NUQ_WORKER_COUNT:-1}
- BULL_AUTH_KEY=${BULL_AUTH_KEY} - BULL_AUTH_KEY=${BULL_AUTH_KEY}
- TEST_API_KEY=${TEST_API_KEY} - TEST_API_KEY=${TEST_API_KEY}
- LOGGING_LEVEL=${LOGGING_LEVEL:-info} - LOGGING_LEVEL=${LOGGING_LEVEL:-info}
@@ -138,7 +139,7 @@ services:
- SEARXNG_CATEGORIES=${SEARXNG_CATEGORIES:-} - SEARXNG_CATEGORIES=${SEARXNG_CATEGORIES:-}
- ALLOW_LOCAL_WEBHOOKS=${ALLOW_LOCAL_WEBHOOKS:-false} - ALLOW_LOCAL_WEBHOOKS=${ALLOW_LOCAL_WEBHOOKS:-false}
- BLOCK_MEDIA=${BLOCK_MEDIA:-true} - BLOCK_MEDIA=${BLOCK_MEDIA:-true}
- HARNESS_STARTUP_TIMEOUT_MS=60000 - HARNESS_STARTUP_TIMEOUT_MS=${HARNESS_STARTUP_TIMEOUT_MS:-180000}
- NUQ_RABBITMQ_URL=amqp://firecrawl-rabbitmq:5672 - NUQ_RABBITMQ_URL=amqp://firecrawl-rabbitmq:5672
command: node dist/src/harness.js --start-docker command: node dist/src/harness.js --start-docker