vault: session notes 2026-07-09
This commit is contained in:
parent
d00b244db3
commit
18a5a7fcad
|
|
@ -91,3 +91,4 @@ claude mcp add firecrawl \
|
|||
## Related
|
||||
|
||||
- [[firecrawl-self-hosted-capabilities]] — feature parity and value assessment
|
||||
- [[firecrawl-usage-guide]] — how to actually call the running instance (REST/CLI/wired-up clients), as opposed to installing it
|
||||
|
|
|
|||
|
|
@ -53,3 +53,4 @@ Docker Compose stack (API + worker + Playwright/Chromium + Redis + Postgres/Rabb
|
|||
## Related
|
||||
|
||||
- [[firecrawl-self-host-setup]] — how to install and wire it to Claude Code / agents
|
||||
- [[firecrawl-usage-guide]] — how to actually call a running instance (REST/CLI/wired-up clients)
|
||||
|
|
|
|||
|
|
@ -0,0 +1,72 @@
|
|||
---
|
||||
type: howto
|
||||
title: "Use the self-hosted Firecrawl instance (any client, any human)"
|
||||
summary: "How to actually call the tailnet-only self-hosted Firecrawl endpoint — REST API, CLI, and the two clients already wired up — as opposed to how to install/deploy it."
|
||||
tags:
|
||||
- type/howto
|
||||
- tool/firecrawl
|
||||
- domain/web-scraping
|
||||
scope: global
|
||||
last_updated: 2026-07-09
|
||||
date: 2026-07-09
|
||||
update_note: experience-driven
|
||||
related:
|
||||
- firecrawl-self-host-setup
|
||||
- firecrawl-self-hosted-capabilities
|
||||
source: ovh-prod
|
||||
---
|
||||
|
||||
# Using the Self-Hosted Firecrawl Instance
|
||||
|
||||
For installing/deploying, see [[firecrawl-self-host-setup]]. For what works vs cloud, see [[firecrawl-self-hosted-capabilities]]. This note is for actually calling it — any human, any client, any language.
|
||||
|
||||
## Endpoint
|
||||
|
||||
- `http://100.100.43.95:3002` — Tailscale tailnet IP of the ovh-prod VPS (`vps-5d79b132`).
|
||||
- `firecrawl.hyperthrive.io:3002` also resolves to the same tailnet IP for convenience — this is **not** a public route, the DNS name doesn't change the fact that the port is bound only to the tailnet interface (confirmed via `ss` — unreachable on the public IP).
|
||||
- Any caller (human or agent) must be joined to the same tailnet to reach it.
|
||||
|
||||
## No auth required
|
||||
|
||||
Self-hosted Firecrawl has no working auth (Supabase auth is cloud-only — see [[firecrawl-self-hosted-capabilities]] for why). POST directly, no API key needed. Security is enforced entirely by the tailnet-only network bind, not by the app — never expose this port beyond the tailnet.
|
||||
|
||||
## Direct REST usage (any language/client)
|
||||
|
||||
- Scrape: `POST /v1/scrape` — `{"url": "...", "formats": ["markdown"]}`
|
||||
- Crawl: `POST /v1/crawl` — `{"url": "..."}` (async, returns a job id; poll `GET /v1/crawl/{id}`)
|
||||
- Map: `POST /v1/map` — `{"url": "..."}` — discover URLs on a site
|
||||
- `/v1/search` and `/v1/extract` are **not reliably available** on this deployment — search needs a SearXNG instance (not deployed here), extract needs `OPENAI_API_KEY` (not currently set in this stack's `.env`).
|
||||
|
||||
```bash
|
||||
curl -s -X POST http://100.100.43.95:3002/v1/scrape \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{"url":"https://example.com","formats":["markdown"]}'
|
||||
```
|
||||
|
||||
## Via firecrawl-cli
|
||||
|
||||
On any machine with the CLI installed and on the tailnet:
|
||||
|
||||
```bash
|
||||
export FIRECRAWL_API_URL=http://100.100.43.95:3002
|
||||
firecrawl scrape https://example.com
|
||||
firecrawl crawl https://example.com
|
||||
firecrawl map https://example.com
|
||||
```
|
||||
|
||||
No `FIRECRAWL_API_KEY` needed once `FIRECRAWL_API_URL` points somewhere other than the cloud default — auth is skipped automatically.
|
||||
|
||||
## Clients already wired up (as of 2026-07-09)
|
||||
|
||||
- **Desktop Claude Code** — official firecrawl plugin + `firecrawl-cli` installed; `FIRECRAWL_API_URL` set via `~/.credentials` (sourced into shell env). Verified working via `firecrawl scrape`.
|
||||
- **hermes01** (Nous Hermes agent on Proxmox) — native Firecrawl provider built into hermes-agent (`plugins/web/firecrawl/provider.py`) reads `FIRECRAWL_API_URL` directly. Set in `~/.hermes/.env` on hermes01, with `web.extract_backend: firecrawl` in `~/.hermes/config.yaml` — deliberately **not** `web.backend` (the shared fallback), which would also route web *search* to firecrawl and break it, since search needs SearXNG which isn't deployed. Verified end-to-end via a one-shot `hermes chat -z` extract call that returned real scraped markdown.
|
||||
- Both reach the endpoint over Tailscale directly — no public exposure, no per-client auth needed.
|
||||
|
||||
## What it's good for / not good for
|
||||
|
||||
Real headless-Chromium JS rendering and multi-page crawl/map are genuine wins over a plain HTTP fetch (e.g. Claude Code's WebFetch). No anti-bot/proxy rotation, no FIRE-1 autonomous browsing, no working `/search` or `/extract` without extra setup (SearXNG / `OPENAI_API_KEY` respectively). Full comparison in [[firecrawl-self-hosted-capabilities]].
|
||||
|
||||
## Related
|
||||
|
||||
- [[firecrawl-self-host-setup]] — install/deploy and env-var reference
|
||||
- [[firecrawl-self-hosted-capabilities]] — feature parity vs cloud, value vs WebFetch
|
||||
Loading…
Reference in New Issue