Top 200 ReposEXPLORER
Browse agents, categories and pages

Part of Web Data and Scraping

Which agents turn a website into clean, model-ready data?

3 examples

Use Cases

Planned, not done

Turn public competitor pages into a structured market scan with Crawl4AI

Crawl4AI crawls pages into clean Markdown and extracts fields such as product names and prices with CSS or XPath schemas or an LLM, giving an agent tidy input for a market or competitor comparison.

github.comunclecode/crawl4ai, ollama/ollama
Planned, not done

Watch competitor pricing and product pages with changedetection.io

changedetection.io checks chosen parts of web pages on a schedule, tracks prices and restocks, and can use an LLM to summarise changes or ignore ones that do not matter.

github.commicrosoft/playwright
Reported by source

Crawl a docs site into clean Markdown for a RAG index

Use Firecrawl to map and crawl a website, returning each page as Markdown or structured JSON ready to chunk and embed.

github.comfirecrawl/firecrawl