Building an AI Scraping Stack

Language models change what is practical in scraping. Extraction that once required a hand-written parser per site can be described in a prompt and generalised across layouts. The trade is that the expensive part of the pipeline moves from engineering time to per-page inference cost.

Designing around that shift is the whole exercise.

The four layers

A workable stack separates concerns.

1. Fetch. A plain HTTP client for static pages, or a call to the site's own data endpoint. This is the cheapest layer by a wide margin, and you should route as much traffic through it as possible. The technique is in Stop Scraping the Page, Find the API Instead.

2. Render fallback. A browser engine for pages that genuinely need JavaScript. Expensive relative to a fetch, so use it as a fallback rather than a default. Options are compared in Browser Automation Tools Compared.

3. Extraction. Either deterministic parsing, an LLM, or a hybrid. This is where the cost decision lives.

4. Egress. Proxies, chosen by target. The network layer is unchanged by everything above, and the selection logic is in Choosing a Proxy for Web Scraping.

Where the cost concentrates

Two places, and they are easy to underestimate.

Inference per page. Sending a full page to a large model for every record is expensive and slow. The token count for a rendered page with navigation, scripts and boilerplate can be enormous, and most of it is irrelevant.

Rendering. A browser costs compute and bandwidth, and for a large crawl that exceeds the proxy bill. The bandwidth case is in What Is Proxy Bandwidth.

The engineering goal follows directly: minimise tokens and minimise renders.

Reducing tokens

Four techniques, in order of effect:

Extract a fragment first. Use a cheap deterministic parse to isolate the region containing your data, then send only that to the model. This alone often cuts tokens by an order of magnitude.

Strip boilerplate. Navigation, footers, scripts and styles add nothing. Readability-style extraction removes most of them, and the approach is in News Scraping with Python.

Use the accessibility tree where you are in a browser. It is far more compact than HTML and organised around what the page actually presents, which is why MCP browser servers use it.

Cache aggressively. If the page has not changed, do not re-infer. A content hash comparison before extraction eliminates most repeat work, which is the change-detection pattern from Residential Proxies for Price Monitoring.

The hybrid that usually wins

The best cost profile is rarely pure LLM or pure parser.

Deterministic first, model as fallback. Parse with selectors, validate the result, and only send the page to a model when validation fails. Over time the fallback rate tells you which sites need attention.

Model first, then codify. Use a model to figure out the extraction for a new site, then convert the working approach into deterministic code. You pay inference once instead of per page.

Model for hard fields only. Extract the structured fields deterministically and use a model for the messy ones, such as deriving a category from free text.

The first pattern is usually the best default, because it keeps the cheap path cheap and confines the expensive path to exceptions.

Keeping output valid

Models produce plausible text, which is a problem when you need a number. Three rules:

Validate with a schema. The model returns a JSON object, and a schema check rejects anything malformed or implausible. The discipline is in Parsing HTML and JSON Reliably.

Do not let the model compute values. Ask it to extract the price text and parse the number in code, so the numeric conversion is deterministic and testable.

Fail loudly on ambiguity. A model that cannot find a field should return null rather than a guess. Instructing it accordingly and validating for it prevents the worst failure mode, which is confident invention.

Keep the raw page for any record whose extraction failed validation. You can re-run extraction later without re-fetching, which saves both bandwidth and time.

Adding the network layer

Egress is orthogonal to the rest, which means you can treat it as configuration rather than architecture.

• Datacenter for public bulk data on undefended targets.
• Residential for defended consumer sites and anything location-dependent.
• Mobile for the hardest targets, used selectively because of cost.

That allocation is the same one described in A Proxy Strategy for E-Commerce Teams, and it applies to AI pipelines identically. Confirm endpoints with the proxy checker.

One caution: an agentic layer that retries freely can burn through proxy bandwidth and inference cost quickly. Cap retries at the pipeline level, not per component.

When to use an agent instead

A scripted pipeline is right for repeated work. An agent with a browser is right for figuring out how to do something once. If you need to discover the data endpoint, understand a multi-step flow, or build the first working version, an agent is faster. See Playwright MCP: Setup and Use Cases.

Use the agent to learn, then write the script. Running an agent per page in production is the expensive end of the trade.

Monitoring the whole stack

Track per layer, because a failure in one looks like a failure in another:

• Fetch success rate, from the network and rendering layers.
• Rendering rate, meaning what fraction of pages needed a browser.
• Fallback rate, meaning what fraction needed a model.
• Validation failure rate, which is your quality signal.
• Cost per successful record, the number that decides whether the design works.

That last metric is the AI-pipeline equivalent of cost per successful page, and it is the one to optimise. The general framework is in Monitoring Scraper Health.