jared
/
SecondBrain
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
SecondBrain/firecrawl-usage-guide.md

4.9 KiB
Raw Blame History

type title summary tags scope last_updated date update_note related source
howto Use the self-hosted Firecrawl instance (any client, any human) How to actually call the tailnet-only self-hosted Firecrawl endpoint — REST API, CLI, and the two clients already wired up — as opposed to how to install/deploy it.
type/howto
tool/firecrawl
domain/web-scraping
global 2026-07-09 2026-07-09 experience-driven
firecrawl-self-host-setup
firecrawl-self-hosted-capabilities
ovh-prod

Using the Self-Hosted Firecrawl Instance

For installing/deploying, see firecrawl-self-host-setup. For what works vs cloud, see firecrawl-self-hosted-capabilities. This note is for actually calling it — any human, any client, any language.

Endpoint

  • http://100.100.43.95:3002 — Tailscale tailnet IP of the ovh-prod VPS (vps-5d79b132).
  • firecrawl.hyperthrive.io:3002 also resolves to the same tailnet IP for convenience — this is not a public route, the DNS name doesn't change the fact that the port is bound only to the tailnet interface (confirmed via ss — unreachable on the public IP).
  • Any caller (human or agent) must be joined to the same tailnet to reach it.

No auth required

Self-hosted Firecrawl has no working auth (Supabase auth is cloud-only — see firecrawl-self-hosted-capabilities for why). POST directly, no API key needed. Security is enforced entirely by the tailnet-only network bind, not by the app — never expose this port beyond the tailnet.

Direct REST usage (any language/client)

  • Scrape: POST /v1/scrape — {"url": "...", "formats": ["markdown"]}
  • Crawl: POST /v1/crawl — {"url": "..."} (async, returns a job id; poll GET /v1/crawl/{id})
  • Map: POST /v1/map — {"url": "..."} — discover URLs on a site
  • /v1/search and /v1/extract are not reliably available on this deployment — search needs a SearXNG instance (not deployed here), extract needs OPENAI_API_KEY (not currently set in this stack's .env).
curl -s -X POST http://100.100.43.95:3002/v1/scrape \
  -H 'Content-Type: application/json' \
  -d '{"url":"https://example.com","formats":["markdown"]}'

Via firecrawl-cli

On any machine with the CLI installed and on the tailnet:

export FIRECRAWL_API_URL=http://100.100.43.95:3002
firecrawl scrape https://example.com
firecrawl crawl https://example.com
firecrawl map https://example.com

No FIRECRAWL_API_KEY needed once FIRECRAWL_API_URL points somewhere other than the cloud default — auth is skipped automatically.

Clients already wired up (as of 2026-07-09)

  • Desktop Claude Code — official firecrawl plugin + firecrawl-cli installed; FIRECRAWL_API_URL set via ~/.credentials (sourced into shell env). Verified working via firecrawl scrape.
  • hermes01 (Nous Hermes agent on Proxmox) — native Firecrawl provider built into hermes-agent (plugins/web/firecrawl/provider.py) reads FIRECRAWL_API_URL directly. Set in ~/.hermes/.env on hermes01, with web.extract_backend: firecrawl in ~/.hermes/config.yaml — deliberately not web.backend (the shared fallback), which would also route web search to firecrawl and break it, since search needs SearXNG which isn't deployed. Verified end-to-end via a one-shot hermes chat -z extract call that returned real scraped markdown.
  • Both reach the endpoint over Tailscale directly — no public exposure, no per-client auth needed.

What it's good for / not good for

Real headless-Chromium JS rendering and multi-page crawl/map are genuine wins over a plain HTTP fetch (e.g. Claude Code's WebFetch). No anti-bot/proxy rotation, no FIRE-1 autonomous browsing, no working /search or /extract without extra setup (SearXNG / OPENAI_API_KEY respectively). Full comparison in firecrawl-self-hosted-capabilities.

Cloud fallback (manual only, no automatic failover)

As of 2026-07-09 the cloud FIRECRAWL_API_KEY (fc-1bfc9a53...) is still valid on a paid subscription, likely to be canceled within a few weeks. Neither firecrawl-cli nor hermes01's native provider has ranked/automatic fallback — FIRECRAWL_API_URL is a single static switch, not a fallback list. If self-hosted is down or misbehaving:

  • CLI / desktop: unset FIRECRAWL_API_URL (or pass --api-url https://api.firecrawl.dev) — the existing FIRECRAWL_API_KEY in ~/.credentials authenticates against cloud with no further setup.
  • hermes01: swap in FIRECRAWL_API_KEY (currently commented out in ~/.hermes/.env) in place of FIRECRAWL_API_URL — the native provider reads the key and hits cloud instead.
  • This is a manual/one-off swap, not scripted health-check-and-failover — building that wasn't judged worth it given the subscription's short remaining life. Revisit if the subscription is kept longer-term.

Related