Crawl4AI: A Hands-On Review

Crawl4AI is an open-source crawler built for the LLM era. Its selling point is that it returns content in a form models can consume, markdown and structured output, rather than raw HTML you then have to clean.

It is a genuinely useful tool in the right context, and a disappointment if you expect it to solve defences.

What it does

At its core it fetches a page, optionally renders it in a browser, extracts the main content, and outputs markdown plus any structured data you have configured. It also handles crawling link graphs and can run LLM-based extraction on the result.

import asyncio
from crawl4ai import AsyncWebCrawler

async def main():
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(url="https://example.com/article")
        print(result.markdown[:2000])

asyncio.run(main())

The appeal is that the output is immediately usable. Markdown rather than HTML means fewer tokens, cleaner chunks and no boilerplate, which is exactly what the token-reduction argument in Building an AI Scraping Stack calls for.

Where it shines

Content extraction for summarisation and retrieval. Feed it a set of article URLs, get clean markdown, chunk and embed. This is its strongest use case and the one most people need.

Sites without heavy defences. On ordinary public pages it is fast to set up and reliable.

Structured extraction with a schema. You can define the fields you want and have the result returned as an object, which saves writing a parser per site.

Crawling a small site. Following links within a domain, with some control over depth and scope, is handled without writing the traversal yourself.

Local control. It is open source and runs on your infrastructure, so there is no per-page API cost and no data leaving your environment. That distinguishes it from managed scraping browsers, compared in Scraping Browsers Explained.

Where it falls short

Defended targets. It is a crawler, not an anti-bot solution. Cloudflare challenges, aggressive fingerprinting and account-gated content are outside what it handles by itself.

Browser dependency weight. The browser-based mode pulls in a headless engine, with the memory and lifecycle considerations described in Puppeteer Web Scraping. Runs accumulate resources if you do not manage them.

Extraction variability. Where LLM-based extraction is used, output varies between runs. That is fine for research and wrong for a pipeline that needs determinism.

Selector control. Relying on its main-content detection is convenient until it picks the wrong region. For precise fields, explicit selectors remain necessary, and the discipline is in Parsing HTML and JSON Reliably.

Rate limiting and pacing. It is not opinionated about politeness. If you crawl with concurrency, that is a decision you made, and it carries the risks in Rate Limiting vs Blocking.

Adding a proxy

This is what turns it from occasionally useful to dependable, because most reliability problems on real targets are network problems.

Proxy support is configured on the crawl request or the browser configuration, depending on the mode.

result = await crawler.arun(
    url="https://example.com/article",
    proxy_config={
        "server": "http://proxy.lightningbytes.com:1080",
        "username": "lb-USERNAME",
        "password": "SECRET",
    },
)

Three notes on doing this well.

Use the right IP type. Residential for defended sites, mobile for the hardest. A datacenter address will produce challenges regardless of how good the extraction is, per Datacenter Proxies: Speed vs Detectability.

Keep concurrency modest. One request at a time per endpoint, with parallelism coming from additional endpoints. The pool design is in What Is a Proxy Pool.

Verify it applied. A misconfigured proxy results in direct requests that look identical to success. Confirm the exit IP rather than assuming, using the proxy checker.

Is it the right tool for you?

Three questions settle it.

Do you need clean text for a model? Then it is a strong fit, and probably faster than assembling the pieces yourself.

Do you need precise, repeatable structured fields at scale? Then write your own parser, or use it for discovery and codify the extraction afterwards.

Is the target defended? Then the crawler choice matters far less than the IP type. Fix the network first; the extraction layer is downstream of it.

A practical setup

Use it in browser mode only where JavaScript requires it, otherwise the lighter HTTP mode. Block images and media, since they cost bandwidth and add nothing, following What Is Proxy Bandwidth. Configure a residential or mobile endpoint, set concurrency to one per endpoint, and add a delay with jitter.

Then measure. Success rate and cost per page are the numbers that matter, and they are the subject of Benchmarking Browser Automation Setups. Track them over time as described in Monitoring Scraper Health so a degradation in the target's defences shows up as a trend rather than a surprise.