Web scraping and crawling optimized for LLMs. Scrape any URL into clean markdown, crawl entire websites, map sitemap structures, search the web with auto-scraping, and extract structured data using schemas or natural language.
View documentationUse Firecrawl when your agent needs clean web content for LLM processing — scraping documentation pages, crawling competitor sites, mapping website structures, or extracting structured data like pricing tables or product specs. Ideal for RAG pipelines, content monitoring agents, research workflows, and any agent that needs web data in a format LLMs can actually use.
Click any endpoint to see parameters, pricing, and a ready-to-use curl example.
Scrape a single URL and return clean, LLM-ready content (markdown, HTML, or structured data).
/services/firecrawl/v1/scrape| Name | Type | Required | Description |
|---|---|---|---|
| url | string | Yes | The URL to scrape |
| formats | string[] | No | Output formats — 'markdown', 'html', 'rawHtml', 'screenshot' (default: ['markdown']) |
| onlyMainContent | boolean | No | Strip navigation, headers, footers — return only the main content (default: true) |
curl -X POST \
"https://api.indiegent.com/services/firecrawl/v1/scrape" \
-H "Authorization: Bearer <your-token>" \
-H "X-Agent-Wallet: my-agent" \
-H "Content-Type: application/json" \
-d '{
"url": "https://docs.firecrawl.dev/introduction"
}'api.indiegent.com/services/firecrawlUSDC · Automatic
Payment protocols handled by IndieGent
Tempo