Privacy choices: we'd like to measure visits anonymously (our own cookie-free counter plus Google Analytics, which sets its own cookies; no ads) and load Beehiiv's newsletter sign-up, which may set its own cookies. Nothing optional runs until you choose. Privacy policy

AgentScout / Tasks / Browsing the web / Scraping and crawling

Scraping and crawling tools for web browsing agents

Tools that turn websites into clean data an agent can use. Of the 221 tools on our browsing the web page, 20 fit this narrower job. By pricing: 15 open source, 2 free or freemium, 1 paid, usage-based or enterprise, 2 pricing on the maker's site. The most-starred open repos here are Scrapling (90,396 stars), Crawl4AI (86,582 stars), Lightpanda (37,901 stars). Most come from the browser/computer use (14), MCP servers (4), research (1) categories.

Updated

Scrapling logo

Scrapling

browser/computer useOpen source

Scrapling is a BSD-3-licensed Python web-scraping framework that covers everything from a single request to a full crawl: fast HTTP fetchers, stealth browsers that can render JavaScript and interact with pages, parsing that adapts when a site's layout changes, and spiders with reusable sessions, proxy rotation, retries, pause and resume and built-in exports. You can use it from Python, the terminal, or an AI agent through its MCP server.

Added

Crawl4AI logo

Crawl4AI

browser/computer useOpen source

Crawl4AI is an Apache-2.0 web crawler that turns pages into clean, LLM-ready Markdown, extracts structured JSON and handles browser automation for RAG pipelines and agents, runnable yourself with Python or Docker. Crawl4AI Cloud offers the same capabilities as a hosted scrape, extract and search API with managed browsers and proxies, plus setup for Claude Code, Cursor and Codex.

Added

Lightpanda logo

Lightpanda

browser/computer useOpen source + cloud

Lightpanda is an AGPL-3.0 headless browser written from scratch in Zig for AI agents, crawlers and test automation, not a Chromium fork. It executes JavaScript and exposes the Chrome DevTools Protocol, so existing Playwright and Puppeteer scripts can connect to it, and its own benchmark of 100 real pages shows about 16x less peak memory and 9x faster runs than headless Chrome. Install it via Homebrew or Docker, or use the hosted Lightpanda cloud.

Added

ScrapeGraphAI logo

ScrapeGraphAI

browser/computer useOpen source + cloud

Python library that uses LLMs and graph logic to build scraping pipelines: describe the data you want and it extracts it from websites or documents.

Added

Stagehand logo

Stagehand

browser/computer useOpen source

Stagehand is Browserbase's MIT-licensed SDK for browser agents: you mix ordinary Playwright-style code with natural-language act, extract and observe calls, so an agent can sign in once, keep the session and pull structured, schema-typed data out of any site. It runs a local browser or Browserbase's cloud, works with any major model, and has quick starts for Claude Code, Codex, Mastra and others.

Added

Moli logo

Moli

browser/computer useOpen source

Lightweight Rust headless browser for AI agents that fetches pages, searches and extracts content, usable via CLI, CDP or WebDriver.

Added

Maxun logo

Maxun

browser/computer useOpen source + cloud

Turns websites into structured APIs with no-code robots for extraction and monitoring.

Added

SeleniumBase MCP logo

SeleniumBase MCP

NewMCP serversOpen sourceLocal only

SeleniumBase MCP is an MCP server that gives agents a way to handle stealthy browser automation, testing, and web-scraping via CDP Mode.

Added

Spider logo

Spider

browser/computer useOpen source + cloud

Fast Rust web crawler for collecting data for agents and LLMs.

Added

webclaw logo

webclaw

browser/computer useOpen source + cloud

Rust web extraction tool that turns sites into clean Markdown or JSON for agents, as a CLI, MCP server and REST API, with a hosted API.

Added

Browserable logo

Browserable

browser/computer useOpen source

Browserable is an MIT-licensed, self-hostable browser automation library for building AI agents that navigate sites, fill in forms, click buttons and extract information, reporting 90.4% on the WebVoyager benchmark. You start it with npx browserable or Docker Compose, add a key for Gemini, OpenAI or Claude and a remote browser provider such as Hyperbrowser or Steel, then run tasks from its dashboard, task API or JavaScript SDK.

Added

Scout (Outcrawl) logo

Scout (Outcrawl)

browser/computer useOpen source + cloud

Open-source browser agent from Outcrawl that opens a real browser and clicks, types and reads pages to finish a task you describe.

Added

WebRouter logo

WebRouter

Newbrowser/computer useOpen source

WebRouter is an Apache-2.0 open standard and SDK from Nimble that gives AI agents one typed interface for web access: search, page extraction, live data, answers, deep research, site-specific scrapers and browser control. It works across many web-data providers with your own keys, routes each task by cost, latency and coverage, and keeps the call and result shape the same when you switch vendors, with governance controls for provider choice, scoped keys and explicit opt-in for writes.

Added

Teracrawl logo

Teracrawl

NewMCP serversOpen sourceNot yet checked

High-performance web crawler API optimized for LLMs. Turn any search or website into clean Markdown using remote browsers. Firecrawl alternative

Added

Qz Browser Agent logo

Qz Browser Agent

NewresearchOpen source

Qz Browser Agent is an open-source agent project that helps builders handle local-first AI agent for Chrome: browser automation, web research, and data extraction.

Added

Apify logo

Apify

browser/computer useFreemium

Marketplace of thousands of ready-to-run scrapers and automation tools ("Actors") for real-time web data, usable from apps and AI agents, including via MCP.

Added

BrowserAct logo

BrowserAct

browser/computer useFreemium

No-code builder for reusable AI web scrapers that collect structured data and connect it to apps, workflows and agents.

Added

ByteFlow logo

ByteFlow

no-code buildersPaid

ByteFlow is a no-code builder for AI agents on top of the systems a company already runs (ERP, CRM, in-house and legacy apps) through APIs or a browser agent. Agents prepare the work, such as a CRM update or ERP entry, and a person approves it with one tap. Includes voice agents in 10+ Indian languages and document extraction.

Added

Clonvo logo

Clonvo

NewMCP serversSee siteNot yet checked

Web search, scraping, crawling, research, a browser agent, and company and people data.

Added

NativePort logo

NativePort

NewMCP serversSee siteNot yet checked

30+ web-access and AI APIs behind one key: search, scraping, browsers, voice, OCR and LLMs.

Added

GitHub repos (16)

RepoGitHub starsLicenceLast push
Scrapling ↗90,396BSD 3-Clause2026-10-29
Crawl4AI ↗86,582Apache License 2.02026-10-26
Lightpanda ↗37,901GNU Affero General Public License v3.02026-10-30
ScrapeGraphAI ↗32,368MIT2026-10-06
Stagehand ↗25,871MIT2026-10-30
Moli ↗21,676—2026-10-30
Maxun ↗18,086GNU AGPL v3.02026-10-28
SeleniumBase MCP ↗13,236——
Spider ↗2,811MIT2026-10-30
webclaw ↗2,410AGPL-3.02026-10-28
Browserable ↗1,209MIT2025-08-27
Scout (Outcrawl) ↗947Apache-2.0, except packages/agenthands which stays MIT2026-10-21
WebRouter ↗570——
Teracrawl ↗289——
Qz Browser Agent ↗214——

Top 15 of 16 by stars. Stars and licence come from each repo, checked when the listing was last verified.

FAQ

What are the best AI tools for browsing the web focused on scraping and crawling?

We list 20. The first three by GitHub stars and featured status are Scrapling: Scrapling is a BSD-3-licensed Python web-scraping framework that covers everything from a single request to a full crawl: fast HTTP fetchers, stealth browsers that can render JavaScript and interact with pages, parsing that adapts when a site's layout changes, and spiders with reusable sessions, proxy rotation, retries, pause and resume and built-in exports. You can use it from Python, the terminal, or an AI agent through its MCP server.; Crawl4AI: Crawl4AI is an Apache-2.0 web crawler that turns pages into clean, LLM-ready Markdown, extracts structured JSON and handles browser automation for RAG pipelines and agents, runnable yourself with Python or Docker. Crawl4AI Cloud offers the same capabilities as a hosted scrape, extract and search API with managed browsers and proxies, plus setup for Claude Code, Cursor and Codex.; Lightpanda: Lightpanda is an AGPL-3.0 headless browser written from scratch in Zig for AI agents, crawlers and test automation, not a Chromium fork. It executes JavaScript and exposes the Chrome DevTools Protocol, so existing Playwright and Puppeteer scripts can connect to it, and its own benchmark of 100 real pages shows about 16x less peak memory and 9x faster runs than headless Chrome. Install it via Homebrew or Docker, or use the hosted Lightpanda cloud.

Are there free or open-source options?

Yes: 15 open source, 2 free or freemium, 1 paid, usage-based or enterprise, 2 pricing on the maker's site. Pricing is as each maker states it.

More from browsing the web

Tools are grouped by what each says it does. Missing one? Submit it free.