Top 200 ReposEXPLORER
Browse agents, categories and pages
Back to results
Planned, not doneThe source describes an intention, not a finished result.

Collect a structured web dataset without knowing the URLs, using Firecrawl’s agent

Firecrawl’s /agent endpoint takes a plain-language request and an optional JSON schema, searches and browses on its own, and returns structured data, with a credit cap on each run.

Try the promptRead the original (opens in a new tab)

The problem

How it works

Firecrawl documents /agent as the replacement for its older extract endpoint. Instead of giving it pages, you describe what you need, such as the founders of a company or a product’s price at several retailers in one country, and optionally pass a JSON schema (Pydantic or Zod in the SDKs) for the shape of the answer. The agent searches, opens and navigates sites in parallel and returns data matching the schema. You can still pass URLs to keep it focused, fix the country it browses from, turn on SafeSearch filtering, set a credit limit per run, receive progress by webhook, and continue a run that hit its limit from where it stopped.

Results and limitations

This is vendor documentation with sample requests and responses, not a reported project, so it is marked planned.

Before you try it

Firecrawl’s docs say that for a single known page, JSON mode on /scrape is cheaper and returns straight away. Set a credit limit on every run, since a run that hits it fails and returns at most a partial result. Results are kept for 24 hours through the API. Spot-check every row against its source page, and respect sites’ terms and personal-data rules when collecting information about people. Results depend on your setup and were not independently reproduced by Top 200 Repos.

Tools and repositories

Tools
Firecrawl API · Python or Node.js SDK, or cURL
Prerequisites
A Firecrawl API key with credits.

Try it yourself: Describe a dataset for a web agent to collect

Fill the fields to adapt the prompt. Empty fields stay as visible [placeholders].

Preview
Find [What to find]. For each result return [Fields for each result], plus the URL of the page where you found each value. Only use information published on the organisation’s own website or an official registry; if a value cannot be found, leave it empty instead of estimating. Stop after [Maximum results] results.

Pass the same fields as a JSON schema and set maxCredits so the run cannot overspend.

Related videos

About firecrawl/firecrawl

About firecrawl/firecrawl

Independent creators. Listing a video implies no endorsement, sponsorship or partnership.

Sources

  1. Agent (opens in a new tab)

    Project documentation · docs.firecrawl.dev · Undated · Source checked 12 Nov 2026, 10:08 UTC

    What this source supports
    • /agent replaces the legacy /extract endpoint; URLs are optional and the prompt is the only required parameter
    • Searches and navigates sites autonomously, processing sources in parallel, with optional JSON schema output
    • Options include focus URLs, a fixed country, SafeSearch, maxCredits, webhooks and continuing a thread after a credit-limit stop
    • For a single known URL, JSON mode on /scrape is cheaper and synchronous
    • Job results are available through the API for 24 hours
Report a problem

Reported by source

Crawl a docs site into clean Markdown for a RAG index

Use Firecrawl to map and crawl a website, returning each page as Markdown or structured JSON ready to chunk and embed.

github.comfirecrawl/firecrawl
Planned, not done

Build a support chatbot that answers from your help docs with Dify

Turn help-centre pages into a Dify knowledge base, route questions with a classifier, answer from retrieved context and embed the bot on your website.

docs.dify.ailanggenius/dify, firecrawl/firecrawl
Planned, not done

Turn public competitor pages into a structured market scan with Crawl4AI

Crawl4AI crawls pages into clean Markdown and extracts fields such as product names and prices with CSS or XPath schemas or an LLM, giving an agent tidy input for a market or competitor comparison.

github.comunclecode/crawl4ai, ollama/ollama
Planned, not done

Watch competitor pricing and product pages with changedetection.io

changedetection.io checks chosen parts of web pages on a schedule, tracks prices and restocks, and can use an LLM to summarise changes or ignore ones that do not matter.

github.commicrosoft/playwright
Reported by source

Answer technical questions by letting async agents run experiments

Simon Willison sends research questions to cloud coding agents such as Claude Code for web and Codex Cloud, which write and run experiments in a dedicated repo and open a PR.

simonwillison.net · 6 Nov 2025anthropics/claude-code, openai/codex
Reported by source

Build a cited Q&A knowledge base over your own documents

Self-host RAGFlow, upload PDFs, Office files and scans, and ask questions that come back with traceable source chunks.

github.cominfiniflow/ragflow