57 lines
4.5 KiB
Markdown
57 lines
4.5 KiB
Markdown
---
|
||
type: reference
|
||
subtype: pattern/framework
|
||
title: "Firecrawl self-hosted — feature parity, value vs WebFetch, and gotchas"
|
||
summary: "What the open-source self-hosted Firecrawl actually delivers vs the cloud product and vs Claude Code's WebFetch — which features work, which are cloud-only (Fire-engine anti-bot, FIRE-1 agent), where the marketing claims break down, and the security/licensing gotchas."
|
||
tags:
|
||
- type/reference
|
||
- tool/firecrawl
|
||
- domain/web-scraping
|
||
- domain/ai-agents
|
||
scope: global
|
||
last_updated: 2026-07-08
|
||
---
|
||
|
||
# Firecrawl Self-Hosted — Capabilities & Value Assessment
|
||
|
||
Research snapshot 2026-07-08. Sources: [SELF_HOST.md](https://github.com/firecrawl/firecrawl/blob/main/SELF_HOST.md), [Self-Host Overhaul v1.5.0](https://github.com/firecrawl/firecrawl/discussions/1222), issues [#2257](https://github.com/firecrawl/firecrawl/issues/2257), [#2881](https://github.com/firecrawl/firecrawl/issues/2881), [mcp-server#126](https://github.com/firecrawl/firecrawl-mcp-server/issues/126).
|
||
|
||
## Feature parity: self-hosted vs cloud
|
||
|
||
| Feature | Self-hosted | Notes |
|
||
|---|---|---|
|
||
| `/scrape` (JS rendering via Playwright) | ✅ | The core value — real headless Chromium |
|
||
| `/crawl`, `/map`, batch scrape | ✅ | Full multi-page crawling works |
|
||
| `/search` | ⚠️ | Only via a self-run SearXNG instance or your own search API key |
|
||
| `/extract` (LLM extraction) | ⚠️ | Works, but requires your own `OPENAI_API_KEY` (OpenAI-compatible/Ollama OK) |
|
||
| Actions (click/scroll), screenshot | ✅ (degraded reliability) | Basic Playwright actions |
|
||
| PDF parsing | ⚠️ | Works; anti-bot-protected PDFs less reliable than cloud |
|
||
| Stealth / anti-bot / residential proxies | ❌ mostly | No Fire-engine. v1.5.0 added `PROXY_*` env vars + `stealthProxy` flag but you must bring your own proxy pool. Self-host fails on aggressively protected sites where cloud succeeds (#2257) |
|
||
| FIRE-1 / `/agent` autonomous browsing | ❌ | Cloud-only |
|
||
| Browser sandbox, change tracking | ❌ / undocumented | Cloud-leaning |
|
||
|
||
## Value vs Claude Code's WebFetch (fact-check of Firecrawl's blog)
|
||
|
||
- **JS rendering — real advantage.** WebFetch does a plain HTTP fetch; JS-only SPAs return empty shells. Firecrawl's headless Chromium fixes this, and it works self-hosted.
|
||
- **Multi-page crawl/map — real advantage**, works self-hosted. WebFetch is one URL per call.
|
||
- **Anti-bot / proxy rotation — cloud-only.** The blog markets this without disclosing that Fire-engine is excluded from self-host. The biggest omission in their comparison.
|
||
- **Token efficiency — mostly marketing.** The circulated "80–94% savings" figures compare Firecrawl markdown vs *raw HTML*, not vs WebFetch's output, which is already markdown-converted by a fast model. No published benchmark compares the real pair. Additionally, the Firecrawl MCP server costs ~10–20k tokens of tool schema per session, which built-in WebFetch doesn't. Firecrawl only wins on tokens if you fetch many pages per session or need pages WebFetch can't render at all.
|
||
- **Practical stance:** self-hosted Firecrawl complements rather than replaces WebFetch. Use Firecrawl for JS-heavy pages, crawls, and structured extraction; keep WebFetch for simple static pages (zero setup, no MCP overhead).
|
||
|
||
## Gotchas
|
||
|
||
- **No real auth in self-host.** Supabase auth can't be configured self-hosted → auth is effectively bypassed (documented behavior, not a bug). Never expose to the public internet; keep LAN/Tailscale/Docker-internal, or add reverse-proxy auth.
|
||
- **MCP server bug:** `firecrawl-mcp-server` demands `FIRECRAWL_API_KEY` even with `FIRECRAWL_API_URL` set to a self-hosted endpoint (mcp-server#126). Workaround: set any non-empty placeholder — the self-hosted server doesn't validate it.
|
||
- **Bull queue admin UI unauthenticated** unless `BULL_AUTH_KEY` set.
|
||
- **License:** AGPL-3.0 core (SDKs/UI MIT). Fine for private self-hosting; source-release obligation only if you offer a modified version as a network service.
|
||
- **Actively maintained** (v1.5.0 self-host overhaul), but self-host permanently trails cloud on anti-bot robustness.
|
||
|
||
## Resource footprint
|
||
|
||
Docker Compose stack (API + worker + Playwright/Chromium + Redis + Postgres/RabbitMQ). Floor ~2 GB RAM; realistic light personal use 4 GB RAM / 2 cores; comfortable 8 GB. Headless Chromium is the biggest consumer. Images total several GB on disk.
|
||
|
||
## Related
|
||
|
||
- [[firecrawl-self-host-setup]] — how to install and wire it to Claude Code / agents
|
||
- [[firecrawl-usage-guide]] — how to actually call a running instance (REST/CLI/wired-up clients)
|