95 lines
5.6 KiB
Markdown
95 lines
5.6 KiB
Markdown
---
|
||
type: howto
|
||
title: "Self-host Firecrawl with Docker Compose and wire it to Claude Code / agents"
|
||
summary: "Steps to run open-source Firecrawl via Docker Compose, the env vars that matter, how to point the firecrawl MCP server (and any REST client) at the self-hosted endpoint, and the security posture required since self-host has no auth."
|
||
tags:
|
||
- type/howto
|
||
- tool/firecrawl
|
||
- tool/docker
|
||
- domain/web-scraping
|
||
scope: global
|
||
last_updated: 2026-07-09
|
||
---
|
||
|
||
# Self-Hosting Firecrawl (Docker Compose)
|
||
|
||
Prereqs: Docker + Compose, 4 GB RAM headroom (2 GB absolute floor), a few GB disk for images. See [[firecrawl-self-hosted-capabilities]] for what you get/don't get.
|
||
|
||
## Install
|
||
|
||
```bash
|
||
git clone https://github.com/firecrawl/firecrawl.git
|
||
cd firecrawl
|
||
cp apps/api/.env.example .env # check SELF_HOST.md for current layout
|
||
docker compose build && docker compose up -d
|
||
# API serves on http://localhost:3002
|
||
```
|
||
|
||
Follow the current [SELF_HOST.md](https://github.com/firecrawl/firecrawl/blob/main/SELF_HOST.md) — the v1.5.0 self-host overhaul reworked the compose file and added a production-ready image; details drift.
|
||
|
||
## Env vars that matter
|
||
|
||
| Var | Why |
|
||
|---|---|
|
||
| `PORT` / `HOST` | Default 3002 / 0.0.0.0 |
|
||
| `OPENAI_API_KEY` | Only needed for `/extract` / JSON LLM extraction; OpenAI-compatible endpoints and Ollama work |
|
||
| `BULL_AUTH_KEY` | Secures the otherwise-open Bull queue admin UI — always set |
|
||
| `POSTGRES_USER/PASSWORD` | Change from dev defaults. **Do NOT change `POSTGRES_DB`** — the `ghcr.io/firecrawl/nuq-postgres` image's pg_cron is hardwired to `cron.database_name=postgres`; any other DB name makes the init script (`010-nuq.sql` `CREATE EXTENSION pg_cron`) fail and the container exit(3), despite SELF_HOST.md telling you to change all three |
|
||
| `PROXY_SERVER`, `PROXY_USERNAME`, `PROXY_PASSWORD` + `stealthProxy` | Bring-your-own proxies (no built-in pool self-hosted) |
|
||
| `MAX_CPU`, `MAX_RAM` | Reject new jobs under load |
|
||
| SearXNG endpoint | Only if you want `/search` |
|
||
|
||
## Security posture (critical)
|
||
|
||
Self-host has **no working auth** (Supabase auth is cloud-only; the "Supabase client not configured" warning means auth is bypassed). Therefore:
|
||
- Bind to LAN/Tailscale/Docker-internal networks only — never the public internet.
|
||
- On a public VPS behind Traefik: do NOT give it a public router; keep it on an internal Docker network only, or add reverse-proxy basic-auth/forward-auth if a route is unavoidable.
|
||
|
||
## Smoke test
|
||
|
||
```bash
|
||
curl -s -X POST http://localhost:3002/v1/scrape \
|
||
-H 'Content-Type: application/json' \
|
||
-d '{"url":"https://example.com","formats":["markdown"]}'
|
||
```
|
||
|
||
## Wiring to Claude Code / agents — CLI + skill (preferred over MCP)
|
||
|
||
The official CLI (`npm i -g firecrawl-cli`, repo github.com/firecrawl/cli) supports self-host cleanly and avoids the ~10–20k-token MCP schema tax:
|
||
|
||
- `export FIRECRAWL_API_URL=http://<host>:3002` (or `--api-url`) — when the URL isn't `https://api.firecrawl.dev`, **auth is automatically skipped**; no key or placeholder needed.
|
||
- Commands: `scrape`, `search`, `map`, `crawl`, `agent`, `interact`, `monitor`.
|
||
- Official Claude Code integration: `firecrawl/firecrawl-claude-plugin` (install via `/plugin`, search "firecrawl") wraps the CLI as skills — it inherits `FIRECRAWL_API_URL`, so it works self-hosted. Requires the CLI installed on the host machine. `firecrawl/skills` is a separate repo for embedding Firecrawl in application code.
|
||
- Sensible routing: Firecrawl first, WebFetch as fallback when Firecrawl fails or isn't needed.
|
||
|
||
## Wiring via MCP (heavier alternative)
|
||
|
||
```bash
|
||
claude mcp add firecrawl \
|
||
-e FIRECRAWL_API_URL=http://<host>:3002 \
|
||
-e FIRECRAWL_API_KEY=self-hosted-placeholder \
|
||
-- npx -y firecrawl-mcp
|
||
```
|
||
|
||
- `FIRECRAWL_API_KEY` must be a non-empty placeholder even for self-host — the MCP server errors without it ([mcp-server#126](https://github.com/firecrawl/firecrawl-mcp-server/issues/126)); the self-hosted API doesn't validate it.
|
||
- MCP schema costs ~10–20k tokens/session — add the server per-project where scraping is actually used, not globally.
|
||
- Any non-MCP agent (e.g. Hermes) can hit the REST API directly (`POST /v1/scrape`, `/v1/crawl`) — same surface as cloud, so official SDKs work by overriding the API URL.
|
||
|
||
## Production deploy notes (verified 2026-07-09 on ovh-prod)
|
||
|
||
- Prebuilt images exist for all three buildable services — `ghcr.io/firecrawl/firecrawl`, `ghcr.io/firecrawl/playwright-service`, `ghcr.io/firecrawl/nuq-postgres` — no need to build on the server. Override cleanly without touching the upstream compose file via `docker-compose.override.yaml` with `build: !reset null` + `image:` per service (compose v2.24+).
|
||
- Upstream compose has **no restart policy** — add `restart: unless-stopped` in the override or the stack won't survive a reboot.
|
||
- Bind the API port to a specific IP in the override with `ports: !override` (e.g. `"100.100.43.95:3002:3002"`) — replaces, not appends, the upstream 0.0.0.0 binding.
|
||
- The FoundationDB services are experimental; with `NUQ_BACKEND` unset, start only what's needed: `docker compose up -d api nuq-postgres`.
|
||
- Idle footprint observed: ~2.8 GiB total (api ~2.4 GiB).
|
||
|
||
## Ops notes
|
||
|
||
- Update: `git pull && docker compose pull && docker compose up -d` (watch the v1.x release notes; compose layout has changed before).
|
||
- Keep WebFetch as the default for simple static pages; route JS-heavy pages, crawls, and structured extraction to Firecrawl.
|
||
|
||
## Related
|
||
|
||
- [[firecrawl-self-hosted-capabilities]] — feature parity and value assessment
|
||
- [[firecrawl-usage-guide]] — how to actually call the running instance (REST/CLI/wired-up clients), as opposed to installing it
|