Handling Pagination in Scrapers

Pagination is where a working scraper becomes a reliable one. Getting the first page is trivial. Getting every page, once, in the right order, and recovering when something fails partway through is the actual work.

The three shapes of pagination

Offset or page numbers. The classic. A parameter such as ?page=2 or ?offset=60&limit=60. Predictable and easy to parallelise, but vulnerable to drift: if items are added between requests, pages shift and you can miss or duplicate records.

Cursor or token. The server returns an opaque token for the next page. More stable under concurrent modification, because the cursor marks a position rather than a count. The downside is that you cannot jump to an arbitrary page, so sequential fetching is required.

Infinite scroll. There is often still a cursor underneath, triggered by scrolling. Find the request rather than driving the scroll, which is the technique in Stop Scraping the Page, Find the API Instead.

Detecting the end

Getting the end condition wrong produces two failures: an infinite loop, or a silently truncated dataset. A truncated dataset is worse because nobody notices.

Signals to use, in order of reliability:

An explicit field. The best endpoints say so directly: has_more: false, next: null, total_pages: 3. Use these when present.

A cursor that stops advancing. If the response returns the same cursor you sent, you are done.

A short page. If you asked for 60 and got 12, that is likely the last page. Treat it as a hint rather than proof, since some servers cap results differently.

Reaching a stated total. If the first response declares 420 items and you have collected 420, stop.

An empty result set, which is unambiguous but wastes one request.

A dangerous pattern is relying solely on "no results returned", because a transient error can masquerade as the end of the data. Distinguish an empty success from a failed request, and never treat a failure as termination.

async def paginate(fetch_page, first_params):
    params = dict(first_params)
    seen_cursors = set()

    while True:
        response = await fetch_page(params)
        if response.status_code != 200:
            raise RuntimeError(f"page fetch failed: {response.status_code}")

        payload = response.json()
        items = payload.get("items", [])
        if not items:
            break

        yield from items

        next_cursor = payload.get("next")
        if not next_cursor or next_cursor in seen_cursors:
            break
        seen_cursors.add(next_cursor)
        params = {"cursor": next_cursor}

The seen_cursors guard is what prevents an infinite loop when a server returns the same cursor indefinitely, which does happen.

Deduplicating across pages

Offset pagination drifts when the underlying list changes mid-crawl. You will see duplicates, and occasionally gaps.

A stable identifier is the fix. Track seen ids and skip repeats.

seen = set()

def emit(item):
    item_id = item.get("id") or item.get("sku") or item.get("url")
    if item_id in seen:
        return None
    seen.add(item_id)
    return item

Where no identifier exists, build one from the fields that define uniqueness, such as name plus price plus location. Keep it deterministic so the same item always hashes the same way.

Gaps are harder, because you cannot detect what you never saw. The mitigation is a coverage check: compare the number of items collected against the total the site reports, and alert on a mismatch. That is the kind of metric worth tracking continuously, per Monitoring Scraper Health.

Ordering and parallelism

With cursor pagination you must fetch sequentially, which caps throughput. With offset pagination you can fan out, subject to the pacing limits in Async Python Scraping Without Breaking Rate Limits.

If you parallelise, keep the same session for a logical unit of work and give each worker its own endpoint, as described in Understanding Proxy Session IDs.

For deep offset pagination, note that some sites cap the number of reachable pages, often at a round number. If results stop at page 100, that is usually a deliberate server limit rather than an error, and you may need to subdivide the query by category or region to reach the whole set. Property portals are a classic example, covered in Residential Proxies for Real-Estate Data.

Resuming after failure

Long crawls fail partway. Checkpoint the cursor or page number and the set of seen ids, then resume from the checkpoint rather than restarting.

Persist the checkpoint after each page, not at the end. A crawl that completes 95 percent and loses its state on a crash has wasted the whole run, and the repetition also costs bandwidth and increases block risk.

Testing pagination properly

A scraper that works on page one tells you almost nothing. Before trusting it:

• Run to the end of a real listing and compare your count against what the site reports.
• Re-run it and confirm you get the same set, not duplicates.
• Introduce a failure mid-crawl and confirm it resumes correctly.
• Test a category with exactly one page, and one where the count is an exact multiple of the page size. Both are common off-by-one cases.

The surrounding reliability practices are in Parsing HTML and JSON Reliably and Monitoring Scraper Health.