Scraping and crawling tools for web browsing agents
Tools that turn websites into clean data an agent can use. Of the 221 tools on our browsing the web page, 20 fit this narrower job. By pricing: 15 open source, 2 free or freemium, 1 paid, usage-based or enterprise, 2 pricing on the maker's site. The most-starred open repos here are Scrapling (90,396 stars), Crawl4AI (86,582 stars), Lightpanda (37,901 stars). Most come from the browser/computer use (14), MCP servers (4), research (1) categories.
Updated
Scrapling
Scrapling is a BSD-3-licensed Python web-scraping framework that covers everything from a single request to a full crawl: fast HTTP fetchers, stealth browsers that can render JavaScript and interact with pages, parsing that adapts when a site's layout changes, and spiders with reusable sessions, proxy rotation, retries, pause and resume and built-in exports. You can use it from Python, the terminal, or an AI agent through its MCP server.
Added
Crawl4AI
Crawl4AI is an Apache-2.0 web crawler that turns pages into clean, LLM-ready Markdown, extracts structured JSON and handles browser automation for RAG pipelines and agents, runnable yourself with Python or Docker. Crawl4AI Cloud offers the same capabilities as a hosted scrape, extract and search API with managed browsers and proxies, plus setup for Claude Code, Cursor and Codex.
Added
Lightpanda
Lightpanda is an AGPL-3.0 headless browser written from scratch in Zig for AI agents, crawlers and test automation, not a Chromium fork. It executes JavaScript and exposes the Chrome DevTools Protocol, so existing Playwright and Puppeteer scripts can connect to it, and its own benchmark of 100 real pages shows about 16x less peak memory and 9x faster runs than headless Chrome. Install it via Homebrew or Docker, or use the hosted Lightpanda cloud.
Added
ScrapeGraphAI
Python library that uses LLMs and graph logic to build scraping pipelines: describe the data you want and it extracts it from websites or documents.
Added
Stagehand
Stagehand is Browserbase's MIT-licensed SDK for browser agents: you mix ordinary Playwright-style code with natural-language act, extract and observe calls, so an agent can sign in once, keep the session and pull structured, schema-typed data out of any site. It runs a local browser or Browserbase's cloud, works with any major model, and has quick starts for Claude Code, Codex, Mastra and others.
Added
Moli
Lightweight Rust headless browser for AI agents that fetches pages, searches and extracts content, usable via CLI, CDP or WebDriver.
Added
Maxun
Turns websites into structured APIs with no-code robots for extraction and monitoring.
Added
SeleniumBase MCP
SeleniumBase MCP is an MCP server that gives agents a way to handle stealthy browser automation, testing, and web-scraping via CDP Mode.
Added
Spider
Fast Rust web crawler for collecting data for agents and LLMs.
Added
webclaw
Rust web extraction tool that turns sites into clean Markdown or JSON for agents, as a CLI, MCP server and REST API, with a hosted API.
Added
Browserable
Browserable is an MIT-licensed, self-hostable browser automation library for building AI agents that navigate sites, fill in forms, click buttons and extract information, reporting 90.4% on the WebVoyager benchmark. You start it with npx browserable or Docker Compose, add a key for Gemini, OpenAI or Claude and a remote browser provider such as Hyperbrowser or Steel, then run tasks from its dashboard, task API or JavaScript SDK.
Added
Scout (Outcrawl)
Open-source browser agent from Outcrawl that opens a real browser and clicks, types and reads pages to finish a task you describe.
Added
WebRouter
WebRouter is an Apache-2.0 open standard and SDK from Nimble that gives AI agents one typed interface for web access: search, page extraction, live data, answers, deep research, site-specific scrapers and browser control. It works across many web-data providers with your own keys, routes each task by cost, latency and coverage, and keeps the call and result shape the same when you switch vendors, with governance controls for provider choice, scoped keys and explicit opt-in for writes.
Added
Teracrawl
High-performance web crawler API optimized for LLMs. Turn any search or website into clean Markdown using remote browsers. Firecrawl alternative
Added
Qz Browser Agent
Qz Browser Agent is an open-source agent project that helps builders handle local-first AI agent for Chrome: browser automation, web research, and data extraction.
Added
Apify
Marketplace of thousands of ready-to-run scrapers and automation tools ("Actors") for real-time web data, usable from apps and AI agents, including via MCP.
Added
BrowserAct
No-code builder for reusable AI web scrapers that collect structured data and connect it to apps, workflows and agents.
Added
ByteFlow
ByteFlow is a no-code builder for AI agents on top of the systems a company already runs (ERP, CRM, in-house and legacy apps) through APIs or a browser agent. Agents prepare the work, such as a CRM update or ERP entry, and a person approves it with one tap. Includes voice agents in 10+ Indian languages and document extraction.
Added
Clonvo
Web search, scraping, crawling, research, a browser agent, and company and people data.
Added
NativePort
30+ web-access and AI APIs behind one key: search, scraping, browsers, voice, OCR and LLMs.
Added
GitHub repos (16)
| Repo | GitHub stars | Licence | Last push |
|---|---|---|---|
| 90,396 | BSD 3-Clause | 2026-10-29 | |
| 86,582 | Apache License 2.0 | 2026-10-26 | |
| 37,901 | GNU Affero General Public License v3.0 | 2026-10-30 | |
| 32,368 | MIT | 2026-10-06 | |
| 25,871 | MIT | 2026-10-30 | |
| 21,676 | — | 2026-10-30 | |
| 18,086 | GNU AGPL v3.0 | 2026-10-28 | |
| 13,236 | — | — | |
| 2,811 | MIT | 2026-10-30 | |
| 2,410 | AGPL-3.0 | 2026-10-28 | |
| 1,209 | MIT | 2025-08-27 | |
| 947 | Apache-2.0, except packages/agenthands which stays MIT | 2026-10-21 | |
| 570 | — | — | |
| 289 | — | — | |
| 214 | — | — |
Top 15 of 16 by stars. Stars and licence come from each repo, checked when the listing was last verified.
FAQ
What are the best AI tools for browsing the web focused on scraping and crawling?
We list 20. The first three by GitHub stars and featured status are Scrapling: Scrapling is a BSD-3-licensed Python web-scraping framework that covers everything from a single request to a full crawl: fast HTTP fetchers, stealth browsers that can render JavaScript and interact with pages, parsing that adapts when a site's layout changes, and spiders with reusable sessions, proxy rotation, retries, pause and resume and built-in exports. You can use it from Python, the terminal, or an AI agent through its MCP server.; Crawl4AI: Crawl4AI is an Apache-2.0 web crawler that turns pages into clean, LLM-ready Markdown, extracts structured JSON and handles browser automation for RAG pipelines and agents, runnable yourself with Python or Docker. Crawl4AI Cloud offers the same capabilities as a hosted scrape, extract and search API with managed browsers and proxies, plus setup for Claude Code, Cursor and Codex.; Lightpanda: Lightpanda is an AGPL-3.0 headless browser written from scratch in Zig for AI agents, crawlers and test automation, not a Chromium fork. It executes JavaScript and exposes the Chrome DevTools Protocol, so existing Playwright and Puppeteer scripts can connect to it, and its own benchmark of 100 real pages shows about 16x less peak memory and 9x faster runs than headless Chrome. Install it via Homebrew or Docker, or use the hosted Lightpanda cloud.
Are there free or open-source options?
Yes: 15 open source, 2 free or freemium, 1 paid, usage-based or enterprise, 2 pricing on the maker's site. Pricing is as each maker states it.
More from browsing the web
Tools are grouped by what each says it does. Missing one? Submit it free.