Privacy choices: we'd like to measure visits anonymously (our own cookie-free counter plus Google Analytics, which sets its own cookies; no ads) and load Beehiiv's newsletter sign-up, which may set its own cookies. Nothing optional runs until you choose. Privacy policy

AgentScout / Popular lists / Free AI web scraping tools

Free AI web scraping tools (2026)

AI web scraping tools that you can use without paying: open-source projects plus tools with a free plan. Here are the top 25 of 39 from the AgentScout directory.

Quick answer: the top picks on this list, ranked by GitHub stars and listing maturity, are Firecrawl, Scrapling and Crawl4AI, out of 25 tools listed below. At a glance: 24 open-source, 25 free or freemium and 22 local or self-hosted across 25 link-checked tools (last verified ).

Verified on : every tool link on this page was checked by our daily link check. No paid placements.

Ranked comparison (top 10)

#ToolPricingBest forOpen sourceRuns locallyGitHub stars
1FirecrawlOpen source + cloudresearch the web, connect tools via MCPYesSelf-hosted196,925
2ScraplingOpen sourceautomate the browser, build an agentYesSelf-hosted90,396
3Crawl4AIOpen sourceautomate the browser, research the webYesSelf-hosted86,582
4LightpandaOpen source + cloudautomate the browser, automate workflowsYesSelf-hosted37,901
5ScrapeGraphAIOpen source + cloudautomate the browser, build an agentYesSelf-hosted32,368
6MoliOpen sourceautomate the browser, research the webYesSelf-hosted21,676
7Douyin TikTok Download APIOpen source + cloudvoice & phone calls, connect tools via MCPYesOffline20,997
8Apify MCP ServerOpen source + cloudresearch the web, connect tools via MCPYesSelf-hosted13,893
9SeleniumBase MCPOpen sourceautomate the browser, automate workflowsYesSelf-hosted13,236
10Social Media Research SkillsOpen sourceresearch the webYesSelf-hosted6,221

Updated

39 matching tools27 open source39 free or with a free tierMost starred: Firecrawl (★ 196,925)

How we picked and ranked: tools are matched by category, the pricing each maker states and what each tool says it does. They are ranked by GitHub stars as recorded on each listing (link checks 2026-11-15); tools without a tracked repo follow, by listing maturity. No paid placements.

  1. Firecrawl logo

    1. Firecrawl

    researchOpen source + cloud★ 196,925 GitHub stars

    Firecrawl is an AGPL-3.0 web data API for AI agents, open source and available as a hosted service: Search returns full page content for web results, Scrape turns any URL into clean Markdown, HTML, screenshots or structured JSON, Crawl and Map cover whole sites, Interact clicks, scrolls and types before extracting, and Parse converts PDFs and DOCX files. It handles proxies, rate limits and JavaScript-heavy pages for you, and connects to any agent or MCP client with a single command.

    Pricing (as stated): Open source + paid cloud · Mature

  2. Scrapling logo

    2. Scrapling

    browser/computer useOpen source★ 90,396 GitHub stars

    Scrapling is a BSD-3-licensed Python web-scraping framework that covers everything from a single request to a full crawl: fast HTTP fetchers, stealth browsers that can render JavaScript and interact with pages, parsing that adapts when a site's layout changes, and spiders with reusable sessions, proxy rotation, retries, pause and resume and built-in exports. You can use it from Python, the terminal, or an AI agent through its MCP server.

    Pricing (as stated): Free and open source (BSD-3-Clause). · Mature

  3. Crawl4AI logo

    3. Crawl4AI

    browser/computer useOpen source★ 86,582 GitHub stars

    Crawl4AI is an Apache-2.0 web crawler that turns pages into clean, LLM-ready Markdown, extracts structured JSON and handles browser automation for RAG pipelines and agents, runnable yourself with Python or Docker. Crawl4AI Cloud offers the same capabilities as a hosted scrape, extract and search API with managed browsers and proxies, plus setup for Claude Code, Cursor and Codex.

    Pricing (as stated): Open source to self-host; Crawl4AI Cloud includes 1,000 free credits a month. · Mature

  4. Lightpanda logo

    4. Lightpanda

    browser/computer useOpen source + cloud★ 37,901 GitHub stars

    Lightpanda is an AGPL-3.0 headless browser written from scratch in Zig for AI agents, crawlers and test automation, not a Chromium fork. It executes JavaScript and exposes the Chrome DevTools Protocol, so existing Playwright and Puppeteer scripts can connect to it, and its own benchmark of 100 real pages shows about 16x less peak memory and 9x faster runs than headless Chrome. Install it via Homebrew or Docker, or use the hosted Lightpanda cloud.

    Pricing (as stated): Open source + cloud · Mature

  5. ScrapeGraphAI logo

    5. ScrapeGraphAI

    browser/computer useOpen source + cloud★ 32,368 GitHub stars

    Python library that uses LLMs and graph logic to build scraping pipelines: describe the data you want and it extracts it from websites or documents.

    Pricing (as stated): Open source + cloud · Mature

  6. Moli logo

    6. Moli

    browser/computer useOpen source★ 21,676 GitHub stars

    Lightweight Rust headless browser for AI agents that fetches pages, searches and extracts content, usable via CLI, CDP or WebDriver.

    Pricing (as stated): Open source · Mature

  7. Douyin TikTok Download API logo

    7. Douyin TikTok Download API

    MCP serversOpen source + cloudNot yet checked★ 20,997 GitHub stars

    、TikTok API, AI agent MCP ,Docker 。 Self-hosted TikTok & Douyin scraper and no-watermark video downloader — async REST API, MCP server for AI agents, CLI and web console, plus offline speech-to-text.

    Pricing (as stated): Open-source (Apache-2.0); see site for any hosted plans · Mature

  8. Apify MCP Server logo

    8. Apify MCP Server

    MCP serversOpen source + cloudNot yet checked★ 13,893 GitHub stars

    Apify's MCP server giving agents access to thousands of scrapers for social media, search and maps.

    Pricing (as stated): Open source + cloud · Mature

  9. SeleniumBase MCP logo

    9. SeleniumBase MCP

    MCP serversOpen sourceLocal only★ 13,236 GitHub stars

    SeleniumBase MCP is an MCP server that gives agents a way to handle stealthy browser automation, testing, and web-scraping via CDP Mode.

    Pricing (as stated): Open source · Mature

  10. Social Media Research Skills logo

    10. Social Media Research Skills

    agent infraOpen source★ 6,221 GitHub stars

    AI agent skills for social media research. Outlier posts, comment mining, competitor teardowns, ad libraries & trends across TikTok, Instagram, YouTube, Reddit, X, LinkedIn & more. Powered by ScrapeCreators. Works with Claude Code, Cursor, Codex, Gemini CLI.

    Pricing (as stated): Open source · Mature

  11. Monid logo

    11. Monid

    agent infraOpen source + cloud★ 6,217 GitHub stars

    Gateway that gives an agent one base URL and one key to call thousands of paid tools, from web scraping and enrichment to image and voice generation, billed per call.

    Pricing (as stated): Open source + cloud · Mature

  12. Wigolo logo

    12. Wigolo

    MCP serversOpen sourceLocal only★ 5,511 GitHub stars

    Local-first web search, fetch and crawl for coding agents over MCP, with no API keys.

    Pricing (as stated): Open source · Mature

  13. Spider logo

    13. Spider

    browser/computer useOpen source + cloud★ 2,811 GitHub stars

    Fast Rust web crawler for collecting data for agents and LLMs.

    Pricing (as stated): Open source + cloud · Active

  14. Bright Data MCP logo

    14. Bright Data MCP

    MCP serversOpen source + cloudLocal only★ 2,696 GitHub stars

    Bright Data's MCP server for web search, scraping and browsing.

    Pricing (as stated): Open source + cloud · Active

  15. Crw logo

    15. Crw

    MCP serversOpen source + cloudLocal only★ 1,197 GitHub stars

    Fast, lightweight Firecrawl/Tavily alternative in Rust. Web scraper, crawler & search API with MCP server for AI agents.

    Pricing (as stated): Open source + cloud · Active

  16. Scout (Outcrawl) logo

    16. Scout (Outcrawl)

    browser/computer useOpen source + cloud★ 947 GitHub stars

    Open-source browser agent from Outcrawl that opens a real browser and clicks, types and reads pages to finish a task you describe.

    Pricing (as stated): Open source + cloud · Active

  17. Scrawl logo

    17. Scrawl

    codingOpen source★ 804 GitHub stars

    Mac app that lets you show a coding agent a problem: press a hotkey, talk, click and draw over your screen or a mirrored iPhone, and Scrawl turns it into transcribed, cropped frames you paste into your agent.

    Pricing (as stated): Free, MIT licence · Active

  18. XActions logo

    18. XActions

    MCP serversOpen sourceLocal only★ 649 GitHub stars

    XActions is an MCP server that gives agents a way to handle x/Twitter automation MCP: scrape, post, schedule, analyze, engage.

    Pricing (as stated): Open source · Active

  19. WebRouter logo

    19. WebRouter

    browser/computer useOpen source★ 570 GitHub stars

    WebRouter is an Apache-2.0 open standard and SDK from Nimble that gives AI agents one typed interface for web access: search, page extraction, live data, answers, deep research, site-specific scrapers and browser control. It works across many web-data providers with your own keys, routes each task by cost, latency and coverage, and keeps the call and result shape the same when you switch vendors, with governance controls for provider choice, scoped keys and explicit opt-in for writes.

    Pricing (as stated): Open source · Active

  20. XCrawl logo

    20. XCrawl

    agent infraFreemium★ 449 GitHub stars

    XCrawl is a web scraping, crawling and SERP API that returns clean JSON, Markdown or screenshots in one request, with agent skills for plugging it into AI agents; free trial with no card.

    Pricing (as stated): Free trial, no card; paid credits · Active

  21. OpenAver logo

    21. OpenAver

    frameworksOpen source★ 354 GitHub stars

    OpenAver is an open-source agent project that helps builders handle desktop app that scrapes JAV metadata and generates NFO + cover art.

    Pricing (as stated): Open source · Active

  22. Teracrawl logo

    22. Teracrawl

    MCP serversOpen sourceNot yet checked★ 289 GitHub stars

    High-performance web crawler API optimized for LLMs. Turn any search or website into clean Markdown using remote browsers. Firecrawl alternative

    Pricing (as stated): Open source · Active

  23. PyScrappy logo

    23. PyScrappy

    MCP serversOpen sourceLocal only★ 265 GitHub stars

    PyScrappy is an MCP server that gives agents a way to handle web-scraping toolkit with 22 tools for structured web data as JSON.

    Pricing (as stated): Open source · Active

  24. Pagesift logo

    24. Pagesift

    MCP serversOpen sourceNot yet checked★ 258 GitHub stars

    Pagesift is an MCP server that gives agents a way to handle web scraper, crawler and document parser for AI agents: URLs, PDFs.

    Pricing (as stated): Open source · Active

  25. ScrapeGraph MCP logo

    25. ScrapeGraph MCP

    MCP serversOpen sourceNot yet checked★ 119 GitHub stars

    ScrapeGraphAI's MCP server for AI-powered web scraping.

    Pricing (as stated): Open source · Active

FAQ

What are the best free AI web scraping tools?

Ranked by tracked GitHub stars, the top free AI web scraping tools in our directory are Firecrawl, Scrapling, Crawl4AI, Lightpanda, ScrapeGraphAI. Stars measure popularity, not fit, so compare a few against your own task.

How many free AI web scraping tools does AgentScout list?

39 tools match. This page shows the top 25. Pricing mix on this page: Open source (14), Open source + cloud (10), Freemium (1).

Are these tools really free?

Pricing is what each maker states on their site. Open-source tools are free to self-host but may need a paid model API key or offer paid cloud plans; freemium tools have a free plan with limits. Check each official site.

Where can I see every tool for scraping websites?

Our task page lists all 64 tools we match to scraping websites.

Related

By role: For researchers · For data analysts

Best-of lists

Related categories

Facts come from our directory listings; pricing is as each maker states it and can change. Check official sites before you choose.