Rank #01 · Daily · Infrastructure
Firecrawl: Open-Source Crawl and Scrape Infrastructure for AI Agents
Scrape, crawl, and map sites into clean markdown (and more) on your own machine or infrastructure — or point the same SDKs at the separate hosted Cloud product when you need search and scale.
License: AGPL-3.0 — modifying and offering as a network service can require sharing corresponding source. Legal review before commercial SaaS embedding.
What it does
Firecrawl is open-source knowledge infrastructure for AI agents: scrape pages, crawl sites, map URL trees, and feed LLM-ready output into agents and apps. GitHub description: "Knowledge infrastructure to make your agents smarter. Search, scrape and 550+ data providers through one API." Site: firecrawl.dev. Docs: docs.firecrawl.dev.
README claims (as verified):
- You can use Firecrawl three ways. This repository holds Firecrawl Local, the self-hosted server (
apps/api), and SDKs that work with all three. Hosted Firecrawl Cloud at firecrawl.dev is a separate product (same API surface for many calls, plus Cloud-only features). - Firecrawl Local (
apps/local): a single binary (install viacargo install firecrawl-local) that scrapes, crawls, and maps on your machine without a database, queue, or container. It can expose an MCP server to agents (firecrawl-local initfor Claude Code, Claude Desktop, Cursor, Windsurf, Gemini CLI, Codex — MCP needs Node.js 22+) and serve the Firecrawl v2 API on localhost (firecrawl-local serve, default port 3002, local routes need no API key). JS-heavy pages use Chrome/Chromium already on the machine; PDFs and Office docs are parsed locally. Without a browser, static pages still work. - Self-hosting: deploy the full server from this repo with Docker Compose (Redis, PostgreSQL, browser service), or use the documented Kubernetes / Helm paths. Alexandria is not included in the self-hosted server — it runs on Firecrawl Cloud.
- Firecrawl Cloud (hosted, not this tree alone): adds scale, search, Alexandria (550+ data providers per README), Interact, Agent, Monitor, and richer scrape formats (structured JSON, summaries, screenshots, browser actions). Local can optionally reach Cloud for search/Alexandria or for scrapes that will not load locally (
FIRECRAWL_CLOUD=offkeeps work on-machine). - Core operations documented for Local and/or Cloud: scrape, crawl, map, batch scrape; Cloud-oriented docs also cover search, Interact, Agent, and Alexandria. SDKs (Python, Node.js, plus Go, Java, Rust, Ruby, .NET, PHP, Elixir per README) poll async jobs and work against Cloud or against Local once you set the API URL to
http://localhost:3002. - By default Firecrawl respects robots.txt. The README states it is the end user's responsibility to respect site policies, privacy policies, and terms of use.
License: the project is primarily AGPL-3.0 (root LICENSE is GNU Affero GPL v3; GitHub SPDX agrees). The README states that SDKs and some UI components are MIT — see LICENSE files in those directories. Do not treat the whole tree as MIT-only, and do not describe this repo as only the hosted Cloud product.
Who it's for
- Teams that want LLM-ready web content (markdown and related formats) for agents, RAG pipelines, or tools — via Local, self-host, or the same SDKs against Cloud.
- Developers who prefer running scrape/crawl on their own machine or infra, with optional MCP wiring into coding agents.
- Groups that may later use Cloud-only features (search, Alexandria, Interact/Agent) without rewriting client code around a different API shape.
Less ideal if you need a pure MIT-licensed core for network-deployed modifications without AGPL obligations, cannot run a local browser or the self-host stack when you refuse Cloud fallbacks, or only want a simple one-off HTML fetcher with no agent/API surface.
Stars / language / license
204,235 stars · TypeScript · AGPL-3.0 (primary; SDKs / some UI components MIT per README)
Data verified: 2026-11-20 09:04 Asia/Shanghai (official Daily Top10 gate; language/license confirmed via GitHub API get_repository + README + LICENSE)
Trending note: +825 stars today (daily Trending window; selection captured 09:04 Asia/Shanghai)
About AGPL-3.0: running Local or self-host for your own use is the common path documented here. If you modify the AGPL-covered server software and let others use it over a network, AGPL-3.0 generally requires offering them the source of your modified version. SDK/UI MIT components follow their own LICENSE files. This is a summary, not legal advice.
Category tags
Infrastructure · web-crawler · scraping · LLM-ready · TypeScript · AGPL-3.0 · MCP · self-hosted · Firecrawl
How it differs from similar projects
Compared with generic scrapers (Beautiful Soup, Scrapy, Playwright scripts). Those are libraries or frameworks you assemble yourself. Firecrawl packages scrape/crawl/map into an API + Local binary + MCP, aimed at LLM-ready markdown and agent workflows.
Compared with browser-automation MCP servers such as chrome-devtools-mcp (#07). chrome-devtools-mcp is the official Chrome DevTools bridge for inspecting and driving a live browser. Firecrawl is a crawl/scrape knowledge API (Local/self-host/Cloud) that turns sites into structured content for agents. Overlap on "open a page"; different primary job.
Compared with hosted-only scrape SaaS. Firecrawl Cloud is a hosted product at firecrawl.dev. This repository is the open-source Local binary, self-hosted server, and SDKs — write both sides clearly; do not collapse them into "only Cloud."
Caveats. Cloud benchmarks (coverage, latency) and Alexandria provider counts are the project's claims, not a TidyHow audit. Respect robots.txt and site terms when scraping. No independent throughput or extraction-quality bake-off is claimed here.