jared
/
SecondBrain
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
SecondBrain/firecrawl-self-hosted-capab...

4.5 KiB
Raw Blame History

type subtype title summary tags scope last_updated
reference pattern/framework Firecrawl self-hosted — feature parity, value vs WebFetch, and gotchas What the open-source self-hosted Firecrawl actually delivers vs the cloud product and vs Claude Code's WebFetch — which features work, which are cloud-only (Fire-engine anti-bot, FIRE-1 agent), where the marketing claims break down, and the security/licensing gotchas.
type/reference
tool/firecrawl
domain/web-scraping
domain/ai-agents
global 2026-07-08

Firecrawl Self-Hosted — Capabilities & Value Assessment

Research snapshot 2026-07-08. Sources: SELF_HOST.md, Self-Host Overhaul v1.5.0, issues #2257, #2881, mcp-server#126.

Feature parity: self-hosted vs cloud

Feature Self-hosted Notes
/scrape (JS rendering via Playwright) ✅ The core value — real headless Chromium
/crawl, /map, batch scrape ✅ Full multi-page crawling works
/search ⚠️ Only via a self-run SearXNG instance or your own search API key
/extract (LLM extraction) ⚠️ Works, but requires your own OPENAI_API_KEY (OpenAI-compatible/Ollama OK)
Actions (click/scroll), screenshot ✅ (degraded reliability) Basic Playwright actions
PDF parsing ⚠️ Works; anti-bot-protected PDFs less reliable than cloud
Stealth / anti-bot / residential proxies ❌ mostly No Fire-engine. v1.5.0 added PROXY_* env vars + stealthProxy flag but you must bring your own proxy pool. Self-host fails on aggressively protected sites where cloud succeeds (#2257)
FIRE-1 / /agent autonomous browsing ❌ Cloud-only
Browser sandbox, change tracking ❌ / undocumented Cloud-leaning

Value vs Claude Code's WebFetch (fact-check of Firecrawl's blog)

  • JS rendering — real advantage. WebFetch does a plain HTTP fetch; JS-only SPAs return empty shells. Firecrawl's headless Chromium fixes this, and it works self-hosted.
  • Multi-page crawl/map — real advantage, works self-hosted. WebFetch is one URL per call.
  • Anti-bot / proxy rotation — cloud-only. The blog markets this without disclosing that Fire-engine is excluded from self-host. The biggest omission in their comparison.
  • Token efficiency — mostly marketing. The circulated "80–94% savings" figures compare Firecrawl markdown vs raw HTML, not vs WebFetch's output, which is already markdown-converted by a fast model. No published benchmark compares the real pair. Additionally, the Firecrawl MCP server costs ~10–20k tokens of tool schema per session, which built-in WebFetch doesn't. Firecrawl only wins on tokens if you fetch many pages per session or need pages WebFetch can't render at all.
  • Practical stance: self-hosted Firecrawl complements rather than replaces WebFetch. Use Firecrawl for JS-heavy pages, crawls, and structured extraction; keep WebFetch for simple static pages (zero setup, no MCP overhead).

Gotchas

  • No real auth in self-host. Supabase auth can't be configured self-hosted → auth is effectively bypassed (documented behavior, not a bug). Never expose to the public internet; keep LAN/Tailscale/Docker-internal, or add reverse-proxy auth.
  • MCP server bug: firecrawl-mcp-server demands FIRECRAWL_API_KEY even with FIRECRAWL_API_URL set to a self-hosted endpoint (mcp-server#126). Workaround: set any non-empty placeholder — the self-hosted server doesn't validate it.
  • Bull queue admin UI unauthenticated unless BULL_AUTH_KEY set.
  • License: AGPL-3.0 core (SDKs/UI MIT). Fine for private self-hosting; source-release obligation only if you offer a modified version as a network service.
  • Actively maintained (v1.5.0 self-host overhaul), but self-host permanently trails cloud on anti-bot robustness.

Resource footprint

Docker Compose stack (API + worker + Playwright/Chromium + Redis + Postgres/RabbitMQ). Floor ~2 GB RAM; realistic light personal use 4 GB RAM / 2 cores; comfortable 8 GB. Headless Chromium is the biggest consumer. Images total several GB on disk.

Related