diff --git a/firecrawl-self-host-setup.md b/firecrawl-self-host-setup.md new file mode 100644 index 0000000..e717d9e --- /dev/null +++ b/firecrawl-self-host-setup.md @@ -0,0 +1,85 @@ +--- +type: howto +title: "Self-host Firecrawl with Docker Compose and wire it to Claude Code / agents" +summary: "Steps to run open-source Firecrawl via Docker Compose, the env vars that matter, how to point the firecrawl MCP server (and any REST client) at the self-hosted endpoint, and the security posture required since self-host has no auth." +tags: + - type/howto + - tool/firecrawl + - tool/docker + - domain/web-scraping +scope: global +last_updated: 2026-07-08 +--- + +# Self-Hosting Firecrawl (Docker Compose) + +Prereqs: Docker + Compose, 4 GB RAM headroom (2 GB absolute floor), a few GB disk for images. See [[firecrawl-self-hosted-capabilities]] for what you get/don't get. + +## Install + +```bash +git clone https://github.com/firecrawl/firecrawl.git +cd firecrawl +cp apps/api/.env.example .env # check SELF_HOST.md for current layout +docker compose build && docker compose up -d +# API serves on http://localhost:3002 +``` + +Follow the current [SELF_HOST.md](https://github.com/firecrawl/firecrawl/blob/main/SELF_HOST.md) — the v1.5.0 self-host overhaul reworked the compose file and added a production-ready image; details drift. + +## Env vars that matter + +| Var | Why | +|---|---| +| `PORT` / `HOST` | Default 3002 / 0.0.0.0 | +| `OPENAI_API_KEY` | Only needed for `/extract` / JSON LLM extraction; OpenAI-compatible endpoints and Ollama work | +| `BULL_AUTH_KEY` | Secures the otherwise-open Bull queue admin UI — always set | +| `POSTGRES_USER/PASSWORD/DB` | Change from dev defaults | +| `PROXY_SERVER`, `PROXY_USERNAME`, `PROXY_PASSWORD` + `stealthProxy` | Bring-your-own proxies (no built-in pool self-hosted) | +| `MAX_CPU`, `MAX_RAM` | Reject new jobs under load | +| SearXNG endpoint | Only if you want `/search` | + +## Security posture (critical) + +Self-host has **no working auth** (Supabase auth is cloud-only; the "Supabase client not configured" warning means auth is bypassed). Therefore: +- Bind to LAN/Tailscale/Docker-internal networks only — never the public internet. +- On a public VPS behind Traefik: do NOT give it a public router; keep it on an internal Docker network only, or add reverse-proxy basic-auth/forward-auth if a route is unavoidable. + +## Smoke test + +```bash +curl -s -X POST http://localhost:3002/v1/scrape \ + -H 'Content-Type: application/json' \ + -d '{"url":"https://example.com","formats":["markdown"]}' +``` + +## Wiring to Claude Code / agents — CLI + skill (preferred over MCP) + +The official CLI (`npm i -g firecrawl-cli`, repo github.com/firecrawl/cli) supports self-host cleanly and avoids the ~10–20k-token MCP schema tax: + +- `export FIRECRAWL_API_URL=http://:3002` (or `--api-url`) — when the URL isn't `https://api.firecrawl.dev`, **auth is automatically skipped**; no key or placeholder needed. +- Commands: `scrape`, `search`, `map`, `crawl`, `agent`, `interact`, `monitor`. +- Official Claude Code integration: `firecrawl/firecrawl-claude-plugin` (install via `/plugin`, search "firecrawl") wraps the CLI as skills — it inherits `FIRECRAWL_API_URL`, so it works self-hosted. Requires the CLI installed on the host machine. `firecrawl/skills` is a separate repo for embedding Firecrawl in application code. +- Sensible routing: Firecrawl first, WebFetch as fallback when Firecrawl fails or isn't needed. + +## Wiring via MCP (heavier alternative) + +```bash +claude mcp add firecrawl \ + -e FIRECRAWL_API_URL=http://:3002 \ + -e FIRECRAWL_API_KEY=self-hosted-placeholder \ + -- npx -y firecrawl-mcp +``` + +- `FIRECRAWL_API_KEY` must be a non-empty placeholder even for self-host — the MCP server errors without it ([mcp-server#126](https://github.com/firecrawl/firecrawl-mcp-server/issues/126)); the self-hosted API doesn't validate it. +- MCP schema costs ~10–20k tokens/session — add the server per-project where scraping is actually used, not globally. +- Any non-MCP agent (e.g. Hermes) can hit the REST API directly (`POST /v1/scrape`, `/v1/crawl`) — same surface as cloud, so official SDKs work by overriding the API URL. + +## Ops notes + +- Update: `git pull && docker compose build && docker compose up -d` (watch the v1.x release notes; compose layout has changed before). +- Keep WebFetch as the default for simple static pages; route JS-heavy pages, crawls, and structured extraction to Firecrawl. + +## Related + +- [[firecrawl-self-hosted-capabilities]] — feature parity and value assessment diff --git a/firecrawl-self-hosted-capabilities.md b/firecrawl-self-hosted-capabilities.md new file mode 100644 index 0000000..9aa32fe --- /dev/null +++ b/firecrawl-self-hosted-capabilities.md @@ -0,0 +1,55 @@ +--- +type: reference +subtype: pattern/framework +title: "Firecrawl self-hosted — feature parity, value vs WebFetch, and gotchas" +summary: "What the open-source self-hosted Firecrawl actually delivers vs the cloud product and vs Claude Code's WebFetch — which features work, which are cloud-only (Fire-engine anti-bot, FIRE-1 agent), where the marketing claims break down, and the security/licensing gotchas." +tags: + - type/reference + - tool/firecrawl + - domain/web-scraping + - domain/ai-agents +scope: global +last_updated: 2026-07-08 +--- + +# Firecrawl Self-Hosted — Capabilities & Value Assessment + +Research snapshot 2026-07-08. Sources: [SELF_HOST.md](https://github.com/firecrawl/firecrawl/blob/main/SELF_HOST.md), [Self-Host Overhaul v1.5.0](https://github.com/firecrawl/firecrawl/discussions/1222), issues [#2257](https://github.com/firecrawl/firecrawl/issues/2257), [#2881](https://github.com/firecrawl/firecrawl/issues/2881), [mcp-server#126](https://github.com/firecrawl/firecrawl-mcp-server/issues/126). + +## Feature parity: self-hosted vs cloud + +| Feature | Self-hosted | Notes | +|---|---|---| +| `/scrape` (JS rendering via Playwright) | ✅ | The core value — real headless Chromium | +| `/crawl`, `/map`, batch scrape | ✅ | Full multi-page crawling works | +| `/search` | ⚠️ | Only via a self-run SearXNG instance or your own search API key | +| `/extract` (LLM extraction) | ⚠️ | Works, but requires your own `OPENAI_API_KEY` (OpenAI-compatible/Ollama OK) | +| Actions (click/scroll), screenshot | ✅ (degraded reliability) | Basic Playwright actions | +| PDF parsing | ⚠️ | Works; anti-bot-protected PDFs less reliable than cloud | +| Stealth / anti-bot / residential proxies | ❌ mostly | No Fire-engine. v1.5.0 added `PROXY_*` env vars + `stealthProxy` flag but you must bring your own proxy pool. Self-host fails on aggressively protected sites where cloud succeeds (#2257) | +| FIRE-1 / `/agent` autonomous browsing | ❌ | Cloud-only | +| Browser sandbox, change tracking | ❌ / undocumented | Cloud-leaning | + +## Value vs Claude Code's WebFetch (fact-check of Firecrawl's blog) + +- **JS rendering — real advantage.** WebFetch does a plain HTTP fetch; JS-only SPAs return empty shells. Firecrawl's headless Chromium fixes this, and it works self-hosted. +- **Multi-page crawl/map — real advantage**, works self-hosted. WebFetch is one URL per call. +- **Anti-bot / proxy rotation — cloud-only.** The blog markets this without disclosing that Fire-engine is excluded from self-host. The biggest omission in their comparison. +- **Token efficiency — mostly marketing.** The circulated "80–94% savings" figures compare Firecrawl markdown vs *raw HTML*, not vs WebFetch's output, which is already markdown-converted by a fast model. No published benchmark compares the real pair. Additionally, the Firecrawl MCP server costs ~10–20k tokens of tool schema per session, which built-in WebFetch doesn't. Firecrawl only wins on tokens if you fetch many pages per session or need pages WebFetch can't render at all. +- **Practical stance:** self-hosted Firecrawl complements rather than replaces WebFetch. Use Firecrawl for JS-heavy pages, crawls, and structured extraction; keep WebFetch for simple static pages (zero setup, no MCP overhead). + +## Gotchas + +- **No real auth in self-host.** Supabase auth can't be configured self-hosted → auth is effectively bypassed (documented behavior, not a bug). Never expose to the public internet; keep LAN/Tailscale/Docker-internal, or add reverse-proxy auth. +- **MCP server bug:** `firecrawl-mcp-server` demands `FIRECRAWL_API_KEY` even with `FIRECRAWL_API_URL` set to a self-hosted endpoint (mcp-server#126). Workaround: set any non-empty placeholder — the self-hosted server doesn't validate it. +- **Bull queue admin UI unauthenticated** unless `BULL_AUTH_KEY` set. +- **License:** AGPL-3.0 core (SDKs/UI MIT). Fine for private self-hosting; source-release obligation only if you offer a modified version as a network service. +- **Actively maintained** (v1.5.0 self-host overhaul), but self-host permanently trails cloud on anti-bot robustness. + +## Resource footprint + +Docker Compose stack (API + worker + Playwright/Chromium + Redis + Postgres/RabbitMQ). Floor ~2 GB RAM; realistic light personal use 4 GB RAM / 2 cores; comfortable 8 GB. Headless Chromium is the biggest consumer. Images total several GB on disk. + +## Related + +- [[firecrawl-self-host-setup]] — how to install and wire it to Claude Code / agents diff --git a/reference/planka-v2-api-gotchas.md b/reference/planka-v2-api-gotchas.md index 8918527..675a27a 100644 --- a/reference/planka-v2-api-gotchas.md +++ b/reference/planka-v2-api-gotchas.md @@ -28,7 +28,20 @@ Directly relevant to the `backlog` CLI in `~/dev/ruby-gems` and any Planka autom and `position`; `listId` alone returns `E_MISSING_OR_INVALID_PARAMS`. 4. **Project delete is not cascading.** `DELETE /api/projects/:id` returns 422 `"Must not have boards"` — delete each board first. -5. **Container uid.** The `ghcr.io/plankanban/planka` image runs as uid 1000 (`node`). +5. **Sails literal-alias routes.** Some delete routes embed a literal parameter-name + segment: card-label detach is `DELETE /api/cards/:cardId/card-labels/labelId::labelId` + (the path literally contains `labelId:` before the id). Normal REST-shaped guesses 404. + When a route 404s unexpectedly, check `server/config/routes.js` in the Planka source. + Related: `GET /api/tasks/:id` doesn't exist (404s to SPA HTML — check Content-Type); + cross-board card move needs `boardId` in the PATCH on top of `listId`+`position`. +6. **Webhooks are instance-level and admin-only.** `GET/POST /api/webhooks` etc.; a + non-admin gets 404 (Sails hides existence, not 403). Not board/project-scoped; + `events` are comma-separated strings in, arrays out. +7. **Attachment download auth differs.** `GET /attachments/:id/download/:filename` + (outside `/api`) authenticates via `Cookie: accessToken=` (or `x-api-key`) — + `Authorization: Bearer` 401s on this route. Also means VCR/webmock setups must + filter Cookie headers, not just Authorization, or the JWT leaks into cassettes. +8. **Container uid.** The `ghcr.io/plankanban/planka` image runs as uid 1000 (`node`). A bind-mounted `/app/data` must be `chown 1000` or attachment/avatar uploads fail (container starts healthy either way — the failure is silent until first upload).