How to crawl websites for AI agents (and pick an AI web crawler)
An AI web crawler fetches pages, renders JavaScript when needed, and returns clean Markdown or JSON a model can read. Pick by job: an open-source library like Crawl4AI for free local crawls, a hosted API like Firecrawl when you want reliability without running browsers, and a browser agent like Browser Use or Stagehand only for logged-in or interactive pages.
Steps
- Decide the output. Markdown for RAG or agent context, typed JSON for a database, or actions for a browser agent. The output decides the tool: a Markdown crawler will not give you a typed schema, and a browser agent is slow, expensive bulk crawling.
- Scope the crawl. List seed URLs or a sitemap, include-patterns, a depth cap, and a page cap. Then check robots.txt and the site's terms before you fetch anything.
- Pick the fetch layer. Plain HTTP is enough for static pages. Reach for a headless browser only when the content needs JavaScript to appear.
- Pick the tool. Use the table below. A hosted API fits when you want reliability without running browsers. A local library fits when the crawl is free and yours to run. A browser agent fits only for logged-in or interactive pages.
- Clean and chunk. Strip navigation, ads, and boilerplate. Chunk by headings. Keep the URL, the page title, the crawl date, and a content hash with every chunk, so you can trace an answer back to its source.
- Treat crawled text as untrusted input. Page text can carry prompt injection: instructions hidden in a page that try to make your agent act. Restrict the seeds you crawl, and never let page text trigger tool calls on its own.
- Give it to an agent. Expose the crawler through an MCP server: Firecrawl MCP, ScrapeGraph MCP, or Playwright MCP. Use a search API such as Tavily or Exa when the agent needs to find pages rather than crawl a known site.
- Recrawl incrementally and spot-check. Compare content hashes and re-embed only the pages that changed. Then spot-check five pages against their source: a clean-looking crawl can still drop a table or merge two sections.
Which tool fits the job
Most top results for this query are single-tool tutorials: a Crawl4AI quickstart, or a cloud walkthrough. This guide is the tool-agnostic version: pick by the job, the license and cost model, and the safety rules, then hook the result into an agent.
| Tool | Best for | Runs where | JS rendering | License or pricing | Agent hookup |
|---|---|---|---|---|---|
| Hosted scraping and crawling that returns clean Markdown or JSON, without running browsers. | Hosted API, or self-host the core with Docker. | Yes, headless browser rendering. | Core AGPL-3.0. Hosted Free plan 1,000 credits a month (about 1,000 pages, no card); Hobby $19 a month for 5,000 credits; Standard $99 a month for 100,000. Scrape and crawl cost 1 credit per page; the JSON format adds 4 per page. Credits do not roll over below Scale. | Firecrawl MCP server, plus SDKs and a REST API. | |
| Free local crawls that turn sites into Markdown for RAG or agent context. | Your machine, as a Python library or a Docker API. | Yes, Playwright rendering. | Apache-2.0. Free to run; you pay only for your machine and, where you use one, the model. | Import it in Python, or call the Docker API from any runtime. | |
| LLM-driven extraction to structured output from a prompt or a schema. | Your machine, as a Python library. | Yes, through Playwright. | MIT. Free to run; the model you call is a separate bill. | ScrapeGraph MCP for agents. | |
| Python crawling where pages change markup and selectors keep breaking. | Your machine, as a Python library. | Yes, browser fetchers alongside the HTTP fetcher. | BSD-3-Clause. Free to run. | Import it in Python; pair it with your own agent code. | |
| High-speed crawling of many pages. | Your machine, as a Rust library or binary. | Yes, optional headless Chrome. | MIT. Free to run. | Call it from Rust, or drive it from your agent's code. | |
| Logged-in or interactive pages, where a crawl cannot go. Not bulk crawling. | Your machine for the libraries; hosted browsers from Browserbase or Steel. | Yes. That is the point: a real browser. | Browser Use MIT, Stagehand MIT, Playwright MCP Apache-2.0, Steel Browser Apache-2.0. You pay for the model and, where you use one, the hosted browser. | Playwright MCP plugs a browser into any MCP client; Browser Use and Stagehand are libraries your agent code calls. | |
| Finding pages when the agent does not already know the URL. | Hosted APIs. | No. They search and return results; they do not render pages. | Hosted, freemium. Check the vendor page for the current free allowance. | REST APIs, and MCP servers for agent clients. |
Model API fees are separate wherever the tool calls a model you pay for.
Browse, submit, or subscribe
Checklist
- Output format chosen: Markdown, typed JSON, or actions.
- Scope set: seed URLs or sitemap, include-patterns, depth and page caps.
- robots.txt and the site's terms checked.
- JavaScript rendering only where the content needs it.
- Every chunk keeps its URL, title, crawl date, and content hash.
- Crawled text treated as untrusted: no tool calls from page text.
- Agent hookup chosen: an MCP server, or a search API for finding pages.
- Recrawl plan set, and five pages spot-checked against their source.
FAQ
What is an AI web crawler?
An AI web crawler fetches pages, renders JavaScript when needed, and returns clean Markdown or JSON a model can read. It is the step between a website and an agent: the crawler turns pages into text the model can use as context or store for retrieval.
Is there a free or open-source AI web crawler?
Yes. Crawl4AI (Apache-2.0), ScrapeGraphAI (MIT), Scrapling (BSD-3-Clause), and Spider (MIT) are free to run on your own machine. Firecrawl's core is open source under AGPL-3.0 and self-hostable, and its hosted plan starts with 1,000 free credits a month.
Is Firecrawl free?
Firecrawl's hosted Free plan gives you 1,000 credits a month, about 1,000 pages, with no card required. Hobby is $19 a month for 5,000 credits and Standard is $99 a month for 100,000. A scrape or crawl costs 1 credit per page, and the JSON format adds 4 credits per page. Credits do not roll over below the Scale plan.
Firecrawl vs Crawl4AI: which should I use?
Use Firecrawl when you want a hosted API and do not want to run browsers yourself. Use Crawl4AI when you want a free local crawl under the permissive Apache-2.0 license and are happy running Python and Playwright. If you self-host Firecrawl's core and build a closed service on it, read the AGPL-3.0 copyleft terms first.
How do I scrape a website with AI?
Decide the output first: Markdown for reading, typed JSON for a database. Then scope the crawl, check robots.txt and the site's terms, and pick the tool from the table below. Clean and chunk the result, keep the source URL and crawl date, and treat the crawled text as untrusted input before you hand it to an agent.
Can an AI agent scrape websites through MCP?
Yes. Firecrawl MCP, ScrapeGraph MCP, and Playwright MCP expose scrape, crawl, and browser tools to any MCP client, so the agent can fetch a page without you writing the crawler. Use a search API such as Tavily or Exa instead when the agent needs to find pages it does not already know.
Is web scraping legal?
There is no single answer, and this is not legal advice. Scraping public data is the usual starting point, but you should respect robots.txt and the site's terms, and rules on personal data vary by country. When in doubt, check the rules where you operate or ask a lawyer.
Related reading
- Scraping and crawling tools for web browsing agents
- Free AI web scraping tools
- Best MCP servers for web search and scraping
- How to automate workflows with AI agents
Scout (Outcrawl)
n8n
When you are ready
Feature details were checked on each vendor's docs or pricing page on . Prices change. Read the vendor page before you buy. No vendor paid for a mention.