Stop Scraping the Page, Find the API Instead

Most modern web pages are assembled in the browser from data fetched over HTTP. The rendered HTML you see is a presentation layer on top of JSON. If you scrape the rendered result, you are parsing a display format and paying for the rendering cost.

Finding the data request instead is usually the single largest improvement available to a scraper.

Why this matters so much

Four practical gains:

Smaller payloads. A JSON response for a product listing is frequently a fraction of the rendered page, especially with images and scripts included. That translates directly into lower bandwidth cost, as we explain in What Is Proxy Bandwidth.

Stable structure. A JSON field name changes less often than a DOM structure and a class name generator. Your parser breaks less.

No browser required. If you have the endpoint, you can use a plain HTTP client rather than a headless browser, which is faster and far lighter than the alternatives in Playwright vs Selenium.

No rendering delay. You skip waiting for the page to finish building, which removes an entire class of flaky waits.

Finding the endpoint

Open the browser's developer tools, go to the Network tab, filter to Fetch or XHR, and reload the page. You are looking for requests returning JSON.

What to look for:

• A response that contains the data you want, recognisable as a JSON structure with the field names you would expect.
• A request with sensible parameters, such as a product id, pagination cursor or page number. That suggests it is the primary data call rather than telemetry.
• A response size that scales with the content, rather than a fixed small payload.

Ignore analytics beacons, tracking pixels and configuration endpoints. The one you want is the one carrying the items.

Replaying the request

Right-click the request and copy it as cURL. That gives you the exact URL, method, headers and body. Then reproduce it in your client.

import requests

response = requests.get(
    "https://example.com/api/v2/products",
    params={"category": "shoes", "page": 1, "per_page": 60},
    headers={
        "Accept": "application/json",
        "Referer": "https://example.com/category/shoes",
        "User-Agent": "Mozilla/5.0 (compatible; research-bot)",
    },
    timeout=(10, 30),
)
data = response.json()

A few things commonly need attention:

The Accept header. Some endpoints return HTML without an explicit JSON accept.

The Referer. A surprising number of endpoints check it.

Query parameters. Copy them exactly. Removing a parameter the server expects can produce an empty result set rather than an error, which is a silent failure worth watching for.

The method. Some data endpoints are POST even when reading, and will reject a GET, which we touch on in Fixing 405 Method Not Allowed.

When it goes wrong

Signed requests. Some endpoints include a signature or hash computed from the parameters plus a secret in the page's JavaScript. Replicating that is possible but brittle, and it is often a sign the operator would prefer you did not use it.

Short-lived tokens. An authorisation header or token that expires quickly means you must refresh it, which usually requires another request or a browser.

CSRF protections. A cookie plus a token header, which must both be present.

Cursor-based pagination with opaque tokens. You must follow the cursor the response gives you rather than constructing page numbers.

When any of these appear, treat it as a signal. The cheap path has been closed deliberately, and the honest response is to reassess rather than escalate.

Deciding when not to

This technique is not a way around access controls, and it should not be used as one. Three limits:

If the endpoint requires authentication you do not have, you do not have access to the data. Finding the URL does not change that.

If the site's terms prohibit collection, the endpoint being callable is irrelevant. We cover the reasoning in Is Web Scraping Legal.

If requests are signed to prove a genuine user session, reproducing that signature is circumventing a control rather than collecting public data.

The legitimate use of this technique is finding the data behind a publicly visible page, which is displayed to anyone who visits it.

A sensible workflow

1. Confirm the data is visible without logging in.
2. Find the data request in the Network tab.
3. Replay it with requests or curl.
4. Validate the shape and record the fields you need.
5. Build pagination using whatever the response provides.
6. Add a fallback to DOM parsing for cases where the endpoint changes, and monitor for it.

That last step is what makes the approach durable. Endpoints change, and a scraper that cannot detect the change fails quietly, which is the problem Monitoring Scraper Health solves.

The comparison it comes from

This is one side of the broader question: use the official API where it exists and is sufficient, use the internal data endpoint when the page is public, and parse the DOM when neither is available. We compare them properly in Web Scraping vs APIs.

Verify an endpoint before scaling with the proxy checker, which reports the exit IP and added latency.