Using the Self-Hosted Firecrawl Instance

For installing/deploying, see firecrawl-self-host-setup. For what works vs cloud, see firecrawl-self-hosted-capabilities. This note is for actually calling it — any human, any client, any language.

Endpoint

http://100.100.43.95:3002 — Tailscale tailnet IP of the ovh-prod VPS (vps-5d79b132).
firecrawl.hyperthrive.io:3002 also resolves to the same tailnet IP for convenience — this is not a public route, the DNS name doesn't change the fact that the port is bound only to the tailnet interface (confirmed via ss — unreachable on the public IP).
Any caller (human or agent) must be joined to the same tailnet to reach it.

No auth required

Self-hosted Firecrawl has no working auth (Supabase auth is cloud-only — see firecrawl-self-hosted-capabilities for why). POST directly, no API key needed. Security is enforced entirely by the tailnet-only network bind, not by the app — never expose this port beyond the tailnet.

Direct REST usage (any language/client)

Scrape: POST /v1/scrape — {"url": "...", "formats": ["markdown"]}
Crawl: POST /v1/crawl — {"url": "..."} (async, returns a job id; poll GET /v1/crawl/{id})
Map: POST /v1/map — {"url": "..."} — discover URLs on a site
/v1/search and /v1/extract are not reliably available on this deployment — search needs a SearXNG instance (not deployed here), extract needs OPENAI_API_KEY (not currently set in this stack's .env).

curl -s -X POST http://100.100.43.95:3002/v1/scrape \
  -H 'Content-Type: application/json' \
  -d '{"url":"https://example.com","formats":["markdown"]}'

Via firecrawl-cli

On any machine with the CLI installed and on the tailnet:

export FIRECRAWL_API_URL=http://100.100.43.95:3002
firecrawl scrape https://example.com
firecrawl crawl https://example.com
firecrawl map https://example.com

No FIRECRAWL_API_KEY needed once FIRECRAWL_API_URL points somewhere other than the cloud default — auth is skipped automatically.

Clients already wired up (as of 2026-07-09)

Desktop Claude Code — official firecrawl plugin + firecrawl-cli installed; FIRECRAWL_API_URL set via ~/.credentials (sourced into shell env). Verified working via firecrawl scrape.
hermes01 (Nous Hermes agent on Proxmox) — native Firecrawl provider built into hermes-agent (plugins/web/firecrawl/provider.py) reads FIRECRAWL_API_URL directly. Set in ~/.hermes/.env on hermes01, with web.extract_backend: firecrawl in ~/.hermes/config.yaml — deliberately not web.backend (the shared fallback), which would also route web search to firecrawl and break it, since search needs SearXNG which isn't deployed. Verified end-to-end via a one-shot hermes chat -z extract call that returned real scraped markdown.
Both reach the endpoint over Tailscale directly — no public exposure, no per-client auth needed.

What it's good for / not good for

Real headless-Chromium JS rendering and multi-page crawl/map are genuine wins over a plain HTTP fetch (e.g. Claude Code's WebFetch). No anti-bot/proxy rotation, no FIRE-1 autonomous browsing, no working /search or /extract without extra setup (SearXNG / OPENAI_API_KEY respectively). Full comparison in firecrawl-self-hosted-capabilities.

Cloud fallback (manual only, no automatic failover)

As of 2026-07-09 the cloud FIRECRAWL_API_KEY (fc-1bfc9a53...) is still valid on a paid subscription, likely to be canceled within a few weeks. Neither firecrawl-cli nor hermes01's native provider has ranked/automatic fallback — FIRECRAWL_API_URL is a single static switch, not a fallback list. If self-hosted is down or misbehaving:

CLI / desktop: unset FIRECRAWL_API_URL (or pass --api-url https://api.firecrawl.dev) — the existing FIRECRAWL_API_KEY in ~/.credentials authenticates against cloud with no further setup.
hermes01: swap in FIRECRAWL_API_KEY (currently commented out in ~/.hermes/.env) in place of FIRECRAWL_API_URL — the native provider reads the key and hits cloud instead.
This is a manual/one-off swap, not scripted health-check-and-failover — building that wasn't judged worth it given the subscription's short remaining life. Revisit if the subscription is kept longer-term.

Floated 2026-07-09, not implemented — a small plugin/skill that routes a given URL/request to one of firecrawl-local, firecrawl-cloud, or WebFetch based on defined criteria, instead of a human/agent picking manually each time. Rough routing criteria to start from:

Simple static page → WebFetch (zero setup, no MCP/CLI overhead)
JS-heavy page / needs multi-page crawl or map → firecrawl-local (self-hosted)
Self-hosted returns an error, times out, or the target needs anti-bot/proxy features self-host lacks (see firecrawl-self-hosted-capabilities) → firecrawl-cloud, if the subscription is still active
Cloud subscription status is the gating factor — this idea is moot once/if the subscription is canceled, unless re-subscribed

This generalizes the manual fallback above into scripted routing logic. Worth revisiting only if: (a) the cloud subscription is kept longer-term, or (b) self-host reliability issues actually recur often enough to justify automation.

firecrawl-self-host-setup — install/deploy and env-var reference
firecrawl-self-hosted-capabilities — feature parity vs cloud, value vs WebFetch

6.0 KiB Raw Blame History