jared
/
SecondBrain
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
SecondBrain/firecrawl-self-host-setup.md

5.6 KiB
Raw Blame History

type title summary tags scope last_updated
howto Self-host Firecrawl with Docker Compose and wire it to Claude Code / agents Steps to run open-source Firecrawl via Docker Compose, the env vars that matter, how to point the firecrawl MCP server (and any REST client) at the self-hosted endpoint, and the security posture required since self-host has no auth.
type/howto
tool/firecrawl
tool/docker
domain/web-scraping
global 2026-07-09

Self-Hosting Firecrawl (Docker Compose)

Prereqs: Docker + Compose, 4 GB RAM headroom (2 GB absolute floor), a few GB disk for images. See firecrawl-self-hosted-capabilities for what you get/don't get.

Install

git clone https://github.com/firecrawl/firecrawl.git
cd firecrawl
cp apps/api/.env.example .env   # check SELF_HOST.md for current layout
docker compose build && docker compose up -d
# API serves on http://localhost:3002

Follow the current SELF_HOST.md — the v1.5.0 self-host overhaul reworked the compose file and added a production-ready image; details drift.

Env vars that matter

Var Why
PORT / HOST Default 3002 / 0.0.0.0
OPENAI_API_KEY Only needed for /extract / JSON LLM extraction; OpenAI-compatible endpoints and Ollama work
BULL_AUTH_KEY Secures the otherwise-open Bull queue admin UI — always set
POSTGRES_USER/PASSWORD Change from dev defaults. Do NOT change POSTGRES_DB — the ghcr.io/firecrawl/nuq-postgres image's pg_cron is hardwired to cron.database_name=postgres; any other DB name makes the init script (010-nuq.sql CREATE EXTENSION pg_cron) fail and the container exit(3), despite SELF_HOST.md telling you to change all three
PROXY_SERVER, PROXY_USERNAME, PROXY_PASSWORD + stealthProxy Bring-your-own proxies (no built-in pool self-hosted)
MAX_CPU, MAX_RAM Reject new jobs under load
SearXNG endpoint Only if you want /search

Security posture (critical)

Self-host has no working auth (Supabase auth is cloud-only; the "Supabase client not configured" warning means auth is bypassed). Therefore:

  • Bind to LAN/Tailscale/Docker-internal networks only — never the public internet.
  • On a public VPS behind Traefik: do NOT give it a public router; keep it on an internal Docker network only, or add reverse-proxy basic-auth/forward-auth if a route is unavoidable.

Smoke test

curl -s -X POST http://localhost:3002/v1/scrape \
  -H 'Content-Type: application/json' \
  -d '{"url":"https://example.com","formats":["markdown"]}'

Wiring to Claude Code / agents — CLI + skill (preferred over MCP)

The official CLI (npm i -g firecrawl-cli, repo github.com/firecrawl/cli) supports self-host cleanly and avoids the ~10–20k-token MCP schema tax:

  • export FIRECRAWL_API_URL=http://<host>:3002 (or --api-url) — when the URL isn't https://api.firecrawl.dev, auth is automatically skipped; no key or placeholder needed.
  • Commands: scrape, search, map, crawl, agent, interact, monitor.
  • Official Claude Code integration: firecrawl/firecrawl-claude-plugin (install via /plugin, search "firecrawl") wraps the CLI as skills — it inherits FIRECRAWL_API_URL, so it works self-hosted. Requires the CLI installed on the host machine. firecrawl/skills is a separate repo for embedding Firecrawl in application code.
  • Sensible routing: Firecrawl first, WebFetch as fallback when Firecrawl fails or isn't needed.

Wiring via MCP (heavier alternative)

claude mcp add firecrawl \
  -e FIRECRAWL_API_URL=http://<host>:3002 \
  -e FIRECRAWL_API_KEY=self-hosted-placeholder \
  -- npx -y firecrawl-mcp
  • FIRECRAWL_API_KEY must be a non-empty placeholder even for self-host — the MCP server errors without it (mcp-server#126); the self-hosted API doesn't validate it.
  • MCP schema costs ~10–20k tokens/session — add the server per-project where scraping is actually used, not globally.
  • Any non-MCP agent (e.g. Hermes) can hit the REST API directly (POST /v1/scrape, /v1/crawl) — same surface as cloud, so official SDKs work by overriding the API URL.

Production deploy notes (verified 2026-07-09 on ovh-prod)

  • Prebuilt images exist for all three buildable services — ghcr.io/firecrawl/firecrawl, ghcr.io/firecrawl/playwright-service, ghcr.io/firecrawl/nuq-postgres — no need to build on the server. Override cleanly without touching the upstream compose file via docker-compose.override.yaml with build: !reset null + image: per service (compose v2.24+).
  • Upstream compose has no restart policy — add restart: unless-stopped in the override or the stack won't survive a reboot.
  • Bind the API port to a specific IP in the override with ports: !override (e.g. "100.100.43.95:3002:3002") — replaces, not appends, the upstream 0.0.0.0 binding.
  • The FoundationDB services are experimental; with NUQ_BACKEND unset, start only what's needed: docker compose up -d api nuq-postgres.
  • Idle footprint observed: ~2.8 GiB total (api ~2.4 GiB).

Ops notes

  • Update: git pull && docker compose pull && docker compose up -d (watch the v1.x release notes; compose layout has changed before).
  • Keep WebFetch as the default for simple static pages; route JS-heavy pages, crawls, and structured extraction to Firecrawl.

Related