AI Potluck
Product / UX / Agent tools & protocols

Firecrawl

Mendable

Web scraping and crawling API built for LLM consumption, turning a URL into clean markdown or structured data while handling JavaScript rendering, pagination and anti-bot measures. It offers scrape, crawl, map, search, batch and agent-extract endpoints, and runs either on infrastructure you operate or as a hosted service.

The self-hosted build cannot take screenshots or run page actions - both need Fire-engine, which is closed and ships in no public repo. That is the gate the openness score rests on. Verified 2026-08-13 via GitHub and the self-hosting guide.

Openness

4 high confidence
4.0
license
AGPL-3.0(OSI, self-host core)
source
public
managed-cloud
firecrawl.dev with 'additional features' beyond OSS = open-core split
core-gated
gated(Fire-engine is closed and its source is in no public repo - the tree ships only a client at apps/api/src/scraper/scrapeURL/engines/fire-engine and .env.example calls FIRE_ENGINE_BETA_URL the fire engine closed beta. Screenshots and page actions are unavailable in the self-hosted stack because both require Fire-engine)

The AGPL-3.0 core is genuinely open, but a piece of the product is withheld from it, which is what keeps this at open core rather than open source. The gate has a name: Fire-engine, Firecrawl's own scraping engine, whose source is in no public repository. What firecrawl/firecrawl publishes under apps/api/src/scraper/scrapeURL/engines/fire-engine is only a client - index.ts, scrape.ts, checkStatus.ts, delete.ts, brandingScript.ts - pointed at a URL you have to be given, and apps/api/.env.example says as much in one line: "# set if you'd like to use the fire engine closed beta" above FIRE_ENGINE_BETA_URL=. The self-hosting guide then spells out what the open build cannot do without it: screenshots and page actions are "Not available in the default stack. Fetch and Playwright both report no support; both require Fire-engine", Fire-engine must be "run and configured separately; it is not included", and the agent, browser, interact and specialty product, audio and video formats need Cloud or another external service. The same .env.example shows the pattern twice more - the product format needs "an external product-extraction service", as do the audio and video formats. These are features of the product itself, kept out of the published source behind a closed beta, so the score stays at 4. The managed cloud advertising "additional features" would not on its own be enough to establish a gate; the evidence here is in the code.

  • https://github.com/mendableai/firecrawl/blob/main/LICENSE recorded 2026-06-04

    AGPL-3.0 license text, Sideguide Technologies Inc.

  • https://www.firecrawl.dev/pricing recorded 2026-06-04

    6 paid managed-cloud tiers (Free/Hobby/Standard/Growth/Scale/Enterprise)

  • https://raw.githubusercontent.com/firecrawl/firecrawl/main/apps/api/.env.example recorded 2026-08-12

    The gate, in the shipped configuration. "# set if you'd like to use the fire engine closed beta" immediately above FIRE_ENGINE_BETA_URL=. Two entries later, the product scrape format "extracts a structured product from a page's HTML via an external product-extraction service; point this at that service's base URL. If unset, requesting the product format returns a warning and no product (same pattern as the audio/video formats and AVGRAB_SERVICE_URL)."

  • https://api.github.com/repos/firecrawl/firecrawl/contents/apps/api/src/scraper/scrapeURL/engines/fire-engine recorded 2026-08-12

    The whole of Fire-engine in the public repo: branding-script, brandingScript.ts, checkStatus.ts, delete.ts, index.ts, scrape.ts. A request/poll/delete client against a remote base URL. The engine it calls is not in this repository or any other public one.

  • https://docs.firecrawl.dev/contributing/self-host recorded 2026-08-12

    What the open build cannot do. Screenshots and page actions: "Not available in the default stack. Fetch and Playwright both report no support; both require Fire-engine." Fire-engine and advanced anti-bot behavior must be "run and configured separately; it is not included." Agent, browser, interact and feedback features and the specialty product, menu, audio and video formats require Firecrawl Cloud or a separate external service.

  • https://api.github.com/repos/firecrawl/firecrawl recorded 2026-08-12

    Repo metadata: license spdx_id AGPL-3.0, default branch main, 165,882 stars, not archived. One license across the tree - the carve-out is not a directory under other terms but an engine that is absent from it.

Adoption

3 low confidence
3.0

Neither the repository nor the pricing page publishes a download, request-volume or customer count of any kind - a paid API sells credits rather than reporting usage - so there is nothing here to band as usage volume. The only honest signal is GitHub stars: 166,863 on the one declared repository, which bands at >10K stars, level 3 on the stars scale, and a stars-based signal caps the level there because a star is not a use. Firecrawl is widely integrated as an agent scraping tool (LangChain among others), so real usage is very likely higher than this band can show. The page the count comes from carries a live star counter, so its content will differ on any later fetch; that is expected drift on this source rather than a signal about the claim.

Capability

4 medium confidence
4.0

Broad scrape and crawl coverage. On the dimensions that matter for a search or scrape tool - coverage, freshness, structured output and rate-limit handling - Firecrawl is strong across all four, and it is among the most capable open scraping tools. Not a 5: there is no benchmark behind it, and the top rung is reserved for products that define their category.

  • https://github.com/mendableai/firecrawl recorded 2026-08-13

    README feature list - scrape, crawl, map, search, agent extract and batch, with LLM-ready output formats; "Covers 96% of the web, including JS-heavy pages"; "P95 latency of 3.4s across millions of pages". The coverage figure links to the vendor's own benchmark post, not to a third party.

Verified 2026-08-12