73 lines
4.0 KiB
Markdown
73 lines
4.0 KiB
Markdown
|
---
|
||
|
|
type: howto
|
||
|
|
title: "Use the self-hosted Firecrawl instance (any client, any human)"
|
||
|
|
summary: "How to actually call the tailnet-only self-hosted Firecrawl endpoint — REST API, CLI, and the two clients already wired up — as opposed to how to install/deploy it."
|
||
|
|
tags:
|
||
|
|
- type/howto
|
||
|
|
- tool/firecrawl
|
||
|
|
- domain/web-scraping
|
||
|
|
scope: global
|
||
|
|
last_updated: 2026-07-09
|
||
|
|
date: 2026-07-09
|
||
|
|
update_note: experience-driven
|
||
|
|
related:
|
||
|
|
- firecrawl-self-host-setup
|
||
|
|
- firecrawl-self-hosted-capabilities
|
||
|
|
source: ovh-prod
|
||
|
|
---
|
||
|
|
|
||
|
|
# Using the Self-Hosted Firecrawl Instance
|
||
|
|
|
||
|
|
For installing/deploying, see [[firecrawl-self-host-setup]]. For what works vs cloud, see [[firecrawl-self-hosted-capabilities]]. This note is for actually calling it — any human, any client, any language.
|
||
|
|
|
||
|
|
## Endpoint
|
||
|
|
|
||
|
|
- `http://100.100.43.95:3002` — Tailscale tailnet IP of the ovh-prod VPS (`vps-5d79b132`).
|
||
|
|
- `firecrawl.hyperthrive.io:3002` also resolves to the same tailnet IP for convenience — this is **not** a public route, the DNS name doesn't change the fact that the port is bound only to the tailnet interface (confirmed via `ss` — unreachable on the public IP).
|
||
|
|
- Any caller (human or agent) must be joined to the same tailnet to reach it.
|
||
|
|
|
||
|
|
## No auth required
|
||
|
|
|
||
|
|
Self-hosted Firecrawl has no working auth (Supabase auth is cloud-only — see [[firecrawl-self-hosted-capabilities]] for why). POST directly, no API key needed. Security is enforced entirely by the tailnet-only network bind, not by the app — never expose this port beyond the tailnet.
|
||
|
|
|
||
|
|
## Direct REST usage (any language/client)
|
||
|
|
|
||
|
|
- Scrape: `POST /v1/scrape` — `{"url": "...", "formats": ["markdown"]}`
|
||
|
|
- Crawl: `POST /v1/crawl` — `{"url": "..."}` (async, returns a job id; poll `GET /v1/crawl/{id}`)
|
||
|
|
- Map: `POST /v1/map` — `{"url": "..."}` — discover URLs on a site
|
||
|
|
- `/v1/search` and `/v1/extract` are **not reliably available** on this deployment — search needs a SearXNG instance (not deployed here), extract needs `OPENAI_API_KEY` (not currently set in this stack's `.env`).
|
||
|
|
|
||
|
|
```bash
|
||
|
|
curl -s -X POST http://100.100.43.95:3002/v1/scrape \
|
||
|
|
-H 'Content-Type: application/json' \
|
||
|
|
-d '{"url":"https://example.com","formats":["markdown"]}'
|
||
|
|
```
|
||
|
|
|
||
|
|
## Via firecrawl-cli
|
||
|
|
|
||
|
|
On any machine with the CLI installed and on the tailnet:
|
||
|
|
|
||
|
|
```bash
|
||
|
|
export FIRECRAWL_API_URL=http://100.100.43.95:3002
|
||
|
|
firecrawl scrape https://example.com
|
||
|
|
firecrawl crawl https://example.com
|
||
|
|
firecrawl map https://example.com
|
||
|
|
```
|
||
|
|
|
||
|
|
No `FIRECRAWL_API_KEY` needed once `FIRECRAWL_API_URL` points somewhere other than the cloud default — auth is skipped automatically.
|
||
|
|
|
||
|
|
## Clients already wired up (as of 2026-07-09)
|
||
|
|
|
||
|
|
- **Desktop Claude Code** — official firecrawl plugin + `firecrawl-cli` installed; `FIRECRAWL_API_URL` set via `~/.credentials` (sourced into shell env). Verified working via `firecrawl scrape`.
|
||
|
|
- **hermes01** (Nous Hermes agent on Proxmox) — native Firecrawl provider built into hermes-agent (`plugins/web/firecrawl/provider.py`) reads `FIRECRAWL_API_URL` directly. Set in `~/.hermes/.env` on hermes01, with `web.extract_backend: firecrawl` in `~/.hermes/config.yaml` — deliberately **not** `web.backend` (the shared fallback), which would also route web *search* to firecrawl and break it, since search needs SearXNG which isn't deployed. Verified end-to-end via a one-shot `hermes chat -z` extract call that returned real scraped markdown.
|
||
|
|
- Both reach the endpoint over Tailscale directly — no public exposure, no per-client auth needed.
|
||
|
|
|
||
|
|
## What it's good for / not good for
|
||
|
|
|
||
|
|
Real headless-Chromium JS rendering and multi-page crawl/map are genuine wins over a plain HTTP fetch (e.g. Claude Code's WebFetch). No anti-bot/proxy rotation, no FIRE-1 autonomous browsing, no working `/search` or `/extract` without extra setup (SearXNG / `OPENAI_API_KEY` respectively). Full comparison in [[firecrawl-self-hosted-capabilities]].
|
||
|
|
|
||
|
|
## Related
|
||
|
|
|
||
|
|
- [[firecrawl-self-host-setup]] — install/deploy and env-var reference
|
||
|
|
- [[firecrawl-self-hosted-capabilities]] — feature parity vs cloud, value vs WebFetch
|