diff --git a/firecrawl-self-host-setup.md b/firecrawl-self-host-setup.md index a8a7c08..4b4bf09 100644 --- a/firecrawl-self-host-setup.md +++ b/firecrawl-self-host-setup.md @@ -91,3 +91,4 @@ claude mcp add firecrawl \ ## Related - [[firecrawl-self-hosted-capabilities]] — feature parity and value assessment +- [[firecrawl-usage-guide]] — how to actually call the running instance (REST/CLI/wired-up clients), as opposed to installing it diff --git a/firecrawl-self-hosted-capabilities.md b/firecrawl-self-hosted-capabilities.md index 9aa32fe..ab525b6 100644 --- a/firecrawl-self-hosted-capabilities.md +++ b/firecrawl-self-hosted-capabilities.md @@ -53,3 +53,4 @@ Docker Compose stack (API + worker + Playwright/Chromium + Redis + Postgres/Rabb ## Related - [[firecrawl-self-host-setup]] — how to install and wire it to Claude Code / agents +- [[firecrawl-usage-guide]] — how to actually call a running instance (REST/CLI/wired-up clients) diff --git a/firecrawl-usage-guide.md b/firecrawl-usage-guide.md new file mode 100644 index 0000000..cfd620d --- /dev/null +++ b/firecrawl-usage-guide.md @@ -0,0 +1,72 @@ +--- +type: howto +title: "Use the self-hosted Firecrawl instance (any client, any human)" +summary: "How to actually call the tailnet-only self-hosted Firecrawl endpoint — REST API, CLI, and the two clients already wired up — as opposed to how to install/deploy it." +tags: + - type/howto + - tool/firecrawl + - domain/web-scraping +scope: global +last_updated: 2026-07-09 +date: 2026-07-09 +update_note: experience-driven +related: + - firecrawl-self-host-setup + - firecrawl-self-hosted-capabilities +source: ovh-prod +--- + +# Using the Self-Hosted Firecrawl Instance + +For installing/deploying, see [[firecrawl-self-host-setup]]. For what works vs cloud, see [[firecrawl-self-hosted-capabilities]]. This note is for actually calling it — any human, any client, any language. + +## Endpoint + +- `http://100.100.43.95:3002` — Tailscale tailnet IP of the ovh-prod VPS (`vps-5d79b132`). +- `firecrawl.hyperthrive.io:3002` also resolves to the same tailnet IP for convenience — this is **not** a public route, the DNS name doesn't change the fact that the port is bound only to the tailnet interface (confirmed via `ss` — unreachable on the public IP). +- Any caller (human or agent) must be joined to the same tailnet to reach it. + +## No auth required + +Self-hosted Firecrawl has no working auth (Supabase auth is cloud-only — see [[firecrawl-self-hosted-capabilities]] for why). POST directly, no API key needed. Security is enforced entirely by the tailnet-only network bind, not by the app — never expose this port beyond the tailnet. + +## Direct REST usage (any language/client) + +- Scrape: `POST /v1/scrape` — `{"url": "...", "formats": ["markdown"]}` +- Crawl: `POST /v1/crawl` — `{"url": "..."}` (async, returns a job id; poll `GET /v1/crawl/{id}`) +- Map: `POST /v1/map` — `{"url": "..."}` — discover URLs on a site +- `/v1/search` and `/v1/extract` are **not reliably available** on this deployment — search needs a SearXNG instance (not deployed here), extract needs `OPENAI_API_KEY` (not currently set in this stack's `.env`). + +```bash +curl -s -X POST http://100.100.43.95:3002/v1/scrape \ + -H 'Content-Type: application/json' \ + -d '{"url":"https://example.com","formats":["markdown"]}' +``` + +## Via firecrawl-cli + +On any machine with the CLI installed and on the tailnet: + +```bash +export FIRECRAWL_API_URL=http://100.100.43.95:3002 +firecrawl scrape https://example.com +firecrawl crawl https://example.com +firecrawl map https://example.com +``` + +No `FIRECRAWL_API_KEY` needed once `FIRECRAWL_API_URL` points somewhere other than the cloud default — auth is skipped automatically. + +## Clients already wired up (as of 2026-07-09) + +- **Desktop Claude Code** — official firecrawl plugin + `firecrawl-cli` installed; `FIRECRAWL_API_URL` set via `~/.credentials` (sourced into shell env). Verified working via `firecrawl scrape`. +- **hermes01** (Nous Hermes agent on Proxmox) — native Firecrawl provider built into hermes-agent (`plugins/web/firecrawl/provider.py`) reads `FIRECRAWL_API_URL` directly. Set in `~/.hermes/.env` on hermes01, with `web.extract_backend: firecrawl` in `~/.hermes/config.yaml` — deliberately **not** `web.backend` (the shared fallback), which would also route web *search* to firecrawl and break it, since search needs SearXNG which isn't deployed. Verified end-to-end via a one-shot `hermes chat -z` extract call that returned real scraped markdown. +- Both reach the endpoint over Tailscale directly — no public exposure, no per-client auth needed. + +## What it's good for / not good for + +Real headless-Chromium JS rendering and multi-page crawl/map are genuine wins over a plain HTTP fetch (e.g. Claude Code's WebFetch). No anti-bot/proxy rotation, no FIRE-1 autonomous browsing, no working `/search` or `/extract` without extra setup (SearXNG / `OPENAI_API_KEY` respectively). Full comparison in [[firecrawl-self-hosted-capabilities]]. + +## Related + +- [[firecrawl-self-host-setup]] — install/deploy and env-var reference +- [[firecrawl-self-hosted-capabilities]] — feature parity vs cloud, value vs WebFetch