Using ChatGPT to Build Scrapers

Language models are genuinely good at the tedious parts of scraping: writing a first selector set, adapting to a changed layout, converting a DOM structure into a schema. They are bad at knowing whether the result is correct.

Treating a model as a fast junior engineer is the right mental model. Useful output, still needs review.

What models are good at

Writing a first parser. Given a page's HTML and a description of the fields, a model produces a working extraction with sensible selector choices in seconds. That is an hour of work compressed into a prompt.

Adapting to a change. When a layout shifts, pasting the new markup and the failing output usually produces a corrected selector quickly.

Generating boilerplate. Pagination loops, retry logic, schema definitions and test scaffolding are all well within a model's competence and tedious to write by hand.

Explaining unfamiliar markup. "What is this page structure and where would the price live?" is a question models answer well from a markup sample.

Translating between libraries. Converting a requests scraper to httpx or a Selenium script to Playwright is mechanical, and models handle it reliably.

What models are bad at

Knowing whether the output is right. A model will confidently produce a selector that matches the wrong element. Only validation catches that, and it is why the discipline in Parsing HTML and JSON Reliably matters more with generated code, not less.

Handling live page behaviour. A model reasons about static markup. Dynamic loading, lazy rendering and challenge pages are outside what it can see, so it will not anticipate them.

Selecting an IP type. Nothing about model capability changes the network question. That is still Choosing a Proxy for Web Scraping.

Respecting a site's terms. A model will happily write a scraper for a site that prohibits scraping. That judgement is yours.

Staying current. Model knowledge of a specific site's current markup is stale by definition, which is why pasting live HTML matters.

Prompts that work

A few patterns produce consistently better output.

Give real markup, not a description. Paste a trimmed sample of the actual HTML. Descriptions produce plausible selectors that do not match reality.

Specify the output schema. Ask for exact field names and types, which makes the result immediately usable and testable.

Ask for defensive parsing. Request explicit handling of missing elements, and null rather than a guess. That instruction alone prevents the most common failure mode.

Request a test alongside the parser. Asking for a small fixture-based test with the parser means you have a regression guard immediately, which is the practice described in Beautiful Soup: Parsing HTML Without the Headache.

Constrain the libraries. Specify which HTTP client and parser you use, so you do not receive code requiring a dependency you do not have.

An example instruction:

Here is a trimmed HTML sample of one product card. Write a Python function
using requests and BeautifulSoup that extracts name, price and url. Return
None for any field that is missing. Prefer stable attributes such as
data-testid over class names. Also write a pytest test using the sample
as a fixture.

That prompt is specific about the parser, the schema, the missing-value behaviour and the selector preference, and it asks for the test. The output is directly usable.

Reviewing generated code

Four things to check every time:

Selector fragility. Does it depend on element order or generated class names? If so, ask for stable attributes instead.

Silent failure. Does a missing element produce None, an exception, or a wrong value? The first is correct.

Schema conformance. Does the output shape match what your pipeline expects?

Scope of access. Does the code fetch more than necessary, or follow links indiscriminately? Generated crawlers sometimes do, which is how honeypots get triggered, as noted in Anti-Scraping Techniques and How to Respond.

Repairing a broken parser

The workflow when a site changes:

1. Confirm the failure is a parse problem rather than a block, using the triage in Fixing 403 Forbidden Errors When Scraping.
2. Fetch a fresh sample of the page and check whether the data is present at all.
3. Paste the new markup and the current parser, and ask for a corrected version plus an explanation of what changed.
4. Update the fixture and run the test.
5. Log the change, because a rising parser-change rate on a target is useful business information about that site's stability.

Step three is where models save the most time. A layout change that would take twenty minutes to diagnose manually takes one prompt, provided you have the markup.

The honest limits

Two things to keep straight.

Generated code needs the same review as human code. A model writing a scraper does not remove your responsibility for what that scraper does, including whether the collection is permitted. The framing is in Data Collection Ethics for Engineering Teams and Is Web Scraping Legal.

For production scale, codify. Using a model to generate a parser once and then running the deterministic result is far cheaper than invoking a model per page. That distinction is the core of Building an AI Scraping Stack, and it is the difference between a useful tool and an expensive habit.

Where it fits in the workflow

Use models for the first version, the repair, and the boilerplate. Use deterministic code for the repeated work. Use validation for everything, because the model cannot tell you whether the number it extracted is the right number.

Verify the endpoint before you scale anything with the proxy checker, and keep the resulting pipeline observable with the practices in Monitoring Scraper Health.