Scraping and crawling tools for research agents
Tools that turn websites into clean data an agent can use. Of the 158 tools on our research page, 13 fit this narrower job. By pricing: 9 open source, 1 free or freemium, 2 paid, usage-based or enterprise, 1 pricing on the maker's site. The most-starred open repos here are Crawl4AI (85,667 stars), ScrapeGraphAI (31,882 stars), Moli (19,349 stars). Most come from the MCP & infra (6), browser/computer use (5), research (2) categories.
Updated
Crawl4AI
Open-source crawler that turns websites into clean, LLM-ready Markdown for agents and RAG pipelines.
Added
ScrapeGraphAI
Python library that uses LLMs and graph logic to build scraping pipelines: describe the data you want and it extracts it from websites or documents.
Added
Moli
Lightweight Rust headless browser for AI agents that fetches pages, searches and extracts content, usable via CLI, CDP or WebDriver.
Added
Maxun
Turns websites into structured APIs with no-code robots for extraction and monitoring.
Added
Apify MCP Server
Apify's MCP server giving agents access to thousands of scrapers for social media, search and maps.
Added
Wigolo
Local-first web search, fetch and crawl for coding agents over MCP, with no API keys.
Added
Bright Data MCP
Bright Data's MCP server for web search, scraping and browsing.
Added
Kagi MCP
Official Kagi MCP server with search and page-extraction tools backed by the Kagi API.
Added
Apify
Marketplace of thousands of ready-to-run scrapers and automation tools ("Actors") for real-time web data, usable from apps and AI agents, including via MCP.
Added
Context.dev
Web data API for AI agents that scrapes pages to Markdown, searches the web and crawls sites.
Added
Firecrawl
Turns websites into clean, LLM-ready data for agents.
Added
Tavily
Search and extraction API tuned for agents.
Added
Xquik
Xquik is an X (Twitter) data service for searching and extracting tweets, replies, followers and profiles, monitoring accounts and posting, over REST, an OAuth MCP server and SDKs; pay-as-you-go from $10 or monthly plans from $20.
Added
GitHub repos (8)
| Repo | GitHub stars | Licence | Last push |
|---|---|---|---|
| Crawl4AI ↗ | 85,667 | Apache License 2.0 | 2026-10-15 |
| ScrapeGraphAI ↗ | 31,882 | MIT | 2026-10-06 |
| Moli ↗ | 19,349 | — | 2026-10-19 |
| Maxun ↗ | 17,875 | GNU AGPL v3.0 | 2026-10-18 |
| Apify MCP Server ↗ | 11,578 | MIT | 2026-10-19 |
| Wigolo ↗ | 5,479 | GNU AGPL v3.0 | 2026-09-27 |
| Bright Data MCP ↗ | 2,678 | MIT | 2026-10-18 |
| Kagi MCP ↗ | 544 | — | 2026-07-07 |
8 repos. Stars and licence come from each repo, checked when the listing was last verified.
FAQ
What are the best AI tools for research focused on scraping and crawling?
We list 13. The first three by GitHub stars and featured status are Crawl4AI: Open-source crawler that turns websites into clean, LLM-ready Markdown for agents and RAG pipelines.; ScrapeGraphAI: Python library that uses LLMs and graph logic to build scraping pipelines: describe the data you want and it extracts it from websites or documents.; Moli: Lightweight Rust headless browser for AI agents that fetches pages, searches and extracts content, usable via CLI, CDP or WebDriver.
Are there free or open-source options?
Yes: 9 open source, 1 free or freemium, 2 paid, usage-based or enterprise, 1 pricing on the maker's site. Pricing is as each maker states it.
More from research
Tools are grouped by what each says it does. Missing one? Submit it free.