vault: session notes 2026-07-08
This commit is contained in:
parent
b5ce2c8ed3
commit
e6576e1a71
|
|
@ -0,0 +1,85 @@
|
|||
---
|
||||
type: howto
|
||||
title: "Self-host Firecrawl with Docker Compose and wire it to Claude Code / agents"
|
||||
summary: "Steps to run open-source Firecrawl via Docker Compose, the env vars that matter, how to point the firecrawl MCP server (and any REST client) at the self-hosted endpoint, and the security posture required since self-host has no auth."
|
||||
tags:
|
||||
- type/howto
|
||||
- tool/firecrawl
|
||||
- tool/docker
|
||||
- domain/web-scraping
|
||||
scope: global
|
||||
last_updated: 2026-07-08
|
||||
---
|
||||
|
||||
# Self-Hosting Firecrawl (Docker Compose)
|
||||
|
||||
Prereqs: Docker + Compose, 4 GB RAM headroom (2 GB absolute floor), a few GB disk for images. See [[firecrawl-self-hosted-capabilities]] for what you get/don't get.
|
||||
|
||||
## Install
|
||||
|
||||
```bash
|
||||
git clone https://github.com/firecrawl/firecrawl.git
|
||||
cd firecrawl
|
||||
cp apps/api/.env.example .env # check SELF_HOST.md for current layout
|
||||
docker compose build && docker compose up -d
|
||||
# API serves on http://localhost:3002
|
||||
```
|
||||
|
||||
Follow the current [SELF_HOST.md](https://github.com/firecrawl/firecrawl/blob/main/SELF_HOST.md) — the v1.5.0 self-host overhaul reworked the compose file and added a production-ready image; details drift.
|
||||
|
||||
## Env vars that matter
|
||||
|
||||
| Var | Why |
|
||||
|---|---|
|
||||
| `PORT` / `HOST` | Default 3002 / 0.0.0.0 |
|
||||
| `OPENAI_API_KEY` | Only needed for `/extract` / JSON LLM extraction; OpenAI-compatible endpoints and Ollama work |
|
||||
| `BULL_AUTH_KEY` | Secures the otherwise-open Bull queue admin UI — always set |
|
||||
| `POSTGRES_USER/PASSWORD/DB` | Change from dev defaults |
|
||||
| `PROXY_SERVER`, `PROXY_USERNAME`, `PROXY_PASSWORD` + `stealthProxy` | Bring-your-own proxies (no built-in pool self-hosted) |
|
||||
| `MAX_CPU`, `MAX_RAM` | Reject new jobs under load |
|
||||
| SearXNG endpoint | Only if you want `/search` |
|
||||
|
||||
## Security posture (critical)
|
||||
|
||||
Self-host has **no working auth** (Supabase auth is cloud-only; the "Supabase client not configured" warning means auth is bypassed). Therefore:
|
||||
- Bind to LAN/Tailscale/Docker-internal networks only — never the public internet.
|
||||
- On a public VPS behind Traefik: do NOT give it a public router; keep it on an internal Docker network only, or add reverse-proxy basic-auth/forward-auth if a route is unavoidable.
|
||||
|
||||
## Smoke test
|
||||
|
||||
```bash
|
||||
curl -s -X POST http://localhost:3002/v1/scrape \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{"url":"https://example.com","formats":["markdown"]}'
|
||||
```
|
||||
|
||||
## Wiring to Claude Code / agents — CLI + skill (preferred over MCP)
|
||||
|
||||
The official CLI (`npm i -g firecrawl-cli`, repo github.com/firecrawl/cli) supports self-host cleanly and avoids the ~10–20k-token MCP schema tax:
|
||||
|
||||
- `export FIRECRAWL_API_URL=http://<host>:3002` (or `--api-url`) — when the URL isn't `https://api.firecrawl.dev`, **auth is automatically skipped**; no key or placeholder needed.
|
||||
- Commands: `scrape`, `search`, `map`, `crawl`, `agent`, `interact`, `monitor`.
|
||||
- Official Claude Code integration: `firecrawl/firecrawl-claude-plugin` (install via `/plugin`, search "firecrawl") wraps the CLI as skills — it inherits `FIRECRAWL_API_URL`, so it works self-hosted. Requires the CLI installed on the host machine. `firecrawl/skills` is a separate repo for embedding Firecrawl in application code.
|
||||
- Sensible routing: Firecrawl first, WebFetch as fallback when Firecrawl fails or isn't needed.
|
||||
|
||||
## Wiring via MCP (heavier alternative)
|
||||
|
||||
```bash
|
||||
claude mcp add firecrawl \
|
||||
-e FIRECRAWL_API_URL=http://<host>:3002 \
|
||||
-e FIRECRAWL_API_KEY=self-hosted-placeholder \
|
||||
-- npx -y firecrawl-mcp
|
||||
```
|
||||
|
||||
- `FIRECRAWL_API_KEY` must be a non-empty placeholder even for self-host — the MCP server errors without it ([mcp-server#126](https://github.com/firecrawl/firecrawl-mcp-server/issues/126)); the self-hosted API doesn't validate it.
|
||||
- MCP schema costs ~10–20k tokens/session — add the server per-project where scraping is actually used, not globally.
|
||||
- Any non-MCP agent (e.g. Hermes) can hit the REST API directly (`POST /v1/scrape`, `/v1/crawl`) — same surface as cloud, so official SDKs work by overriding the API URL.
|
||||
|
||||
## Ops notes
|
||||
|
||||
- Update: `git pull && docker compose build && docker compose up -d` (watch the v1.x release notes; compose layout has changed before).
|
||||
- Keep WebFetch as the default for simple static pages; route JS-heavy pages, crawls, and structured extraction to Firecrawl.
|
||||
|
||||
## Related
|
||||
|
||||
- [[firecrawl-self-hosted-capabilities]] — feature parity and value assessment
|
||||
|
|
@ -0,0 +1,55 @@
|
|||
---
|
||||
type: reference
|
||||
subtype: pattern/framework
|
||||
title: "Firecrawl self-hosted — feature parity, value vs WebFetch, and gotchas"
|
||||
summary: "What the open-source self-hosted Firecrawl actually delivers vs the cloud product and vs Claude Code's WebFetch — which features work, which are cloud-only (Fire-engine anti-bot, FIRE-1 agent), where the marketing claims break down, and the security/licensing gotchas."
|
||||
tags:
|
||||
- type/reference
|
||||
- tool/firecrawl
|
||||
- domain/web-scraping
|
||||
- domain/ai-agents
|
||||
scope: global
|
||||
last_updated: 2026-07-08
|
||||
---
|
||||
|
||||
# Firecrawl Self-Hosted — Capabilities & Value Assessment
|
||||
|
||||
Research snapshot 2026-07-08. Sources: [SELF_HOST.md](https://github.com/firecrawl/firecrawl/blob/main/SELF_HOST.md), [Self-Host Overhaul v1.5.0](https://github.com/firecrawl/firecrawl/discussions/1222), issues [#2257](https://github.com/firecrawl/firecrawl/issues/2257), [#2881](https://github.com/firecrawl/firecrawl/issues/2881), [mcp-server#126](https://github.com/firecrawl/firecrawl-mcp-server/issues/126).
|
||||
|
||||
## Feature parity: self-hosted vs cloud
|
||||
|
||||
| Feature | Self-hosted | Notes |
|
||||
|---|---|---|
|
||||
| `/scrape` (JS rendering via Playwright) | ✅ | The core value — real headless Chromium |
|
||||
| `/crawl`, `/map`, batch scrape | ✅ | Full multi-page crawling works |
|
||||
| `/search` | ⚠️ | Only via a self-run SearXNG instance or your own search API key |
|
||||
| `/extract` (LLM extraction) | ⚠️ | Works, but requires your own `OPENAI_API_KEY` (OpenAI-compatible/Ollama OK) |
|
||||
| Actions (click/scroll), screenshot | ✅ (degraded reliability) | Basic Playwright actions |
|
||||
| PDF parsing | ⚠️ | Works; anti-bot-protected PDFs less reliable than cloud |
|
||||
| Stealth / anti-bot / residential proxies | ❌ mostly | No Fire-engine. v1.5.0 added `PROXY_*` env vars + `stealthProxy` flag but you must bring your own proxy pool. Self-host fails on aggressively protected sites where cloud succeeds (#2257) |
|
||||
| FIRE-1 / `/agent` autonomous browsing | ❌ | Cloud-only |
|
||||
| Browser sandbox, change tracking | ❌ / undocumented | Cloud-leaning |
|
||||
|
||||
## Value vs Claude Code's WebFetch (fact-check of Firecrawl's blog)
|
||||
|
||||
- **JS rendering — real advantage.** WebFetch does a plain HTTP fetch; JS-only SPAs return empty shells. Firecrawl's headless Chromium fixes this, and it works self-hosted.
|
||||
- **Multi-page crawl/map — real advantage**, works self-hosted. WebFetch is one URL per call.
|
||||
- **Anti-bot / proxy rotation — cloud-only.** The blog markets this without disclosing that Fire-engine is excluded from self-host. The biggest omission in their comparison.
|
||||
- **Token efficiency — mostly marketing.** The circulated "80–94% savings" figures compare Firecrawl markdown vs *raw HTML*, not vs WebFetch's output, which is already markdown-converted by a fast model. No published benchmark compares the real pair. Additionally, the Firecrawl MCP server costs ~10–20k tokens of tool schema per session, which built-in WebFetch doesn't. Firecrawl only wins on tokens if you fetch many pages per session or need pages WebFetch can't render at all.
|
||||
- **Practical stance:** self-hosted Firecrawl complements rather than replaces WebFetch. Use Firecrawl for JS-heavy pages, crawls, and structured extraction; keep WebFetch for simple static pages (zero setup, no MCP overhead).
|
||||
|
||||
## Gotchas
|
||||
|
||||
- **No real auth in self-host.** Supabase auth can't be configured self-hosted → auth is effectively bypassed (documented behavior, not a bug). Never expose to the public internet; keep LAN/Tailscale/Docker-internal, or add reverse-proxy auth.
|
||||
- **MCP server bug:** `firecrawl-mcp-server` demands `FIRECRAWL_API_KEY` even with `FIRECRAWL_API_URL` set to a self-hosted endpoint (mcp-server#126). Workaround: set any non-empty placeholder — the self-hosted server doesn't validate it.
|
||||
- **Bull queue admin UI unauthenticated** unless `BULL_AUTH_KEY` set.
|
||||
- **License:** AGPL-3.0 core (SDKs/UI MIT). Fine for private self-hosting; source-release obligation only if you offer a modified version as a network service.
|
||||
- **Actively maintained** (v1.5.0 self-host overhaul), but self-host permanently trails cloud on anti-bot robustness.
|
||||
|
||||
## Resource footprint
|
||||
|
||||
Docker Compose stack (API + worker + Playwright/Chromium + Redis + Postgres/RabbitMQ). Floor ~2 GB RAM; realistic light personal use 4 GB RAM / 2 cores; comfortable 8 GB. Headless Chromium is the biggest consumer. Images total several GB on disk.
|
||||
|
||||
## Related
|
||||
|
||||
- [[firecrawl-self-host-setup]] — how to install and wire it to Claude Code / agents
|
||||
|
|
@ -28,7 +28,20 @@ Directly relevant to the `backlog` CLI in `~/dev/ruby-gems` and any Planka autom
|
|||
and `position`; `listId` alone returns `E_MISSING_OR_INVALID_PARAMS`.
|
||||
4. **Project delete is not cascading.** `DELETE /api/projects/:id` returns 422
|
||||
`"Must not have boards"` — delete each board first.
|
||||
5. **Container uid.** The `ghcr.io/plankanban/planka` image runs as uid 1000 (`node`).
|
||||
5. **Sails literal-alias routes.** Some delete routes embed a literal parameter-name
|
||||
segment: card-label detach is `DELETE /api/cards/:cardId/card-labels/labelId::labelId`
|
||||
(the path literally contains `labelId:` before the id). Normal REST-shaped guesses 404.
|
||||
When a route 404s unexpectedly, check `server/config/routes.js` in the Planka source.
|
||||
Related: `GET /api/tasks/:id` doesn't exist (404s to SPA HTML — check Content-Type);
|
||||
cross-board card move needs `boardId` in the PATCH on top of `listId`+`position`.
|
||||
6. **Webhooks are instance-level and admin-only.** `GET/POST /api/webhooks` etc.; a
|
||||
non-admin gets 404 (Sails hides existence, not 403). Not board/project-scoped;
|
||||
`events` are comma-separated strings in, arrays out.
|
||||
7. **Attachment download auth differs.** `GET /attachments/:id/download/:filename`
|
||||
(outside `/api`) authenticates via `Cookie: accessToken=<jwt>` (or `x-api-key`) —
|
||||
`Authorization: Bearer` 401s on this route. Also means VCR/webmock setups must
|
||||
filter Cookie headers, not just Authorization, or the JWT leaks into cassettes.
|
||||
8. **Container uid.** The `ghcr.io/plankanban/planka` image runs as uid 1000 (`node`).
|
||||
A bind-mounted `/app/data` must be `chown 1000` or attachment/avatar uploads fail
|
||||
(container starts healthy either way — the failure is silent until first upload).
|
||||
|
||||
|
|
|
|||
Loading…
Reference in New Issue