From c3d670727f4f0301c7e95b329cf5b29a2c6658ea Mon Sep 17 00:00:00 2001 From: jared Date: Thu, 9 Jul 2026 10:11:09 -0400 Subject: [PATCH] vault: session notes 2026-07-09 --- firecrawl-self-host-setup.md | 14 +++++++++++--- journal/2026-07-09.md | 7 +++++++ 2 files changed, 18 insertions(+), 3 deletions(-) diff --git a/firecrawl-self-host-setup.md b/firecrawl-self-host-setup.md index e717d9e..a8a7c08 100644 --- a/firecrawl-self-host-setup.md +++ b/firecrawl-self-host-setup.md @@ -8,7 +8,7 @@ tags: - tool/docker - domain/web-scraping scope: global -last_updated: 2026-07-08 +last_updated: 2026-07-09 --- # Self-Hosting Firecrawl (Docker Compose) @@ -34,7 +34,7 @@ Follow the current [SELF_HOST.md](https://github.com/firecrawl/firecrawl/blob/ma | `PORT` / `HOST` | Default 3002 / 0.0.0.0 | | `OPENAI_API_KEY` | Only needed for `/extract` / JSON LLM extraction; OpenAI-compatible endpoints and Ollama work | | `BULL_AUTH_KEY` | Secures the otherwise-open Bull queue admin UI — always set | -| `POSTGRES_USER/PASSWORD/DB` | Change from dev defaults | +| `POSTGRES_USER/PASSWORD` | Change from dev defaults. **Do NOT change `POSTGRES_DB`** — the `ghcr.io/firecrawl/nuq-postgres` image's pg_cron is hardwired to `cron.database_name=postgres`; any other DB name makes the init script (`010-nuq.sql` `CREATE EXTENSION pg_cron`) fail and the container exit(3), despite SELF_HOST.md telling you to change all three | | `PROXY_SERVER`, `PROXY_USERNAME`, `PROXY_PASSWORD` + `stealthProxy` | Bring-your-own proxies (no built-in pool self-hosted) | | `MAX_CPU`, `MAX_RAM` | Reject new jobs under load | | SearXNG endpoint | Only if you want `/search` | @@ -75,9 +75,17 @@ claude mcp add firecrawl \ - MCP schema costs ~10–20k tokens/session — add the server per-project where scraping is actually used, not globally. - Any non-MCP agent (e.g. Hermes) can hit the REST API directly (`POST /v1/scrape`, `/v1/crawl`) — same surface as cloud, so official SDKs work by overriding the API URL. +## Production deploy notes (verified 2026-07-09 on ovh-prod) + +- Prebuilt images exist for all three buildable services — `ghcr.io/firecrawl/firecrawl`, `ghcr.io/firecrawl/playwright-service`, `ghcr.io/firecrawl/nuq-postgres` — no need to build on the server. Override cleanly without touching the upstream compose file via `docker-compose.override.yaml` with `build: !reset null` + `image:` per service (compose v2.24+). +- Upstream compose has **no restart policy** — add `restart: unless-stopped` in the override or the stack won't survive a reboot. +- Bind the API port to a specific IP in the override with `ports: !override` (e.g. `"100.100.43.95:3002:3002"`) — replaces, not appends, the upstream 0.0.0.0 binding. +- The FoundationDB services are experimental; with `NUQ_BACKEND` unset, start only what's needed: `docker compose up -d api nuq-postgres`. +- Idle footprint observed: ~2.8 GiB total (api ~2.4 GiB). + ## Ops notes -- Update: `git pull && docker compose build && docker compose up -d` (watch the v1.x release notes; compose layout has changed before). +- Update: `git pull && docker compose pull && docker compose up -d` (watch the v1.x release notes; compose layout has changed before). - Keep WebFetch as the default for simple static pages; route JS-heavy pages, crawls, and structured extraction to Firecrawl. ## Related diff --git a/journal/2026-07-09.md b/journal/2026-07-09.md index f697b51..427f3bf 100644 --- a/journal/2026-07-09.md +++ b/journal/2026-07-09.md @@ -19,3 +19,10 @@ tags: [scope/global, type/log] /home/jared/Documents/SecondBrain/firecrawl-self-host-setup.md /home/jared/Documents/SecondBrain/firecrawl-self-host-setup.md /home/jared/Documents/SecondBrain/forgejo-triage-labels.md + +## Session — 2026-07-09T12:01:42Z + +**Project:** /home/jared/servers/ovh-prod +**Reason:** prompt_input_exit +**Vault notes touched:** +(none)