Rate Limiting vs Blocking: Reading the Signals

The most common reason a scraper gets blocked permanently is that it treated a request to slow down as an obstacle to push through. Rate limiting and blocking are different responses with different remedies, and conflating them escalates the first into the second.

This is how to read the signals and react correctly.

The spectrum of responses

A server has several ways to discourage you, roughly in order of severity.

Silent throttling. Requests succeed but take longer. No status code signals a problem, which makes this the hardest to notice and the easiest to misattribute to proxy latency.

429 Too Many Requests. Explicit and polite. Often accompanied by Retry-After. This is a request to slow down, not a refusal.

Challenge pages. An interstitial that requires JavaScript or interaction. The site is checking whether you are a browser, not deciding you are unwelcome.

403 Forbidden. An explicit refusal. Commonly the result of network classification or behaviour that has already crossed a threshold.

Tarpits. Deliberately slow responses, aiming to make collection uneconomical. Discussed in Anti-Scraping Techniques and How to Respond.

Hard blocks. Connection refused, or responses with no body at all. The address or the ASN is blocked outright.

429 is not a failure

This is the central point. A 429 means the server is willing to serve you, just not at this rate. The correct response is to slow down and honour whatever guidance the response includes.

def handle(response):
    if response.status_code == 429:
        wait = float(response.headers.get("Retry-After", 30))
        time.sleep(wait)
        return "retry"
    if response.status_code == 403:
        return "stop_and_reassess"

Retrying a 429 immediately is the single most common way to convert a rate limit into a block. The server told you what it wanted, and the response was to ignore it.

Reading Retry-After correctly

The header accepts two formats: a number of seconds, or an HTTP date. Handle both, because a date-valued header ignored as invalid falls back to a default that may be far too short.

from email.utils import parsedate_to_datetime
from datetime import datetime, timezone

def retry_after(response, default=30):
    value = response.headers.get("Retry-After")
    if not value:
        return default
    if value.isdigit():
        return int(value)
    try:
        return max(1, int((parsedate_to_datetime(value) - datetime.now(timezone.utc)).total_seconds()))
    except Exception:
        return default

When the header is absent, treat the response as a signal to slow the whole pipeline rather than only pausing the one request. A per-request sleep while other workers continue hammering is not a slowdown.

Telling a rate limit from a block

The distinguishing evidence is in the history.

The "first request" row is the clearest tell. A challenge or 403 on the very first request, before you have done anything, is about the network identity rather than your behaviour. That is the situation described in Datacenter Proxies: Speed vs Detectability.

Why slowing down is usually the right answer

Once you are rate limited, the fastest route back to full throughput is to comply. Three reasons:

The limit usually lifts. Rate limits are frequently time-windowed. Waiting out the window restores access, while hammering extends it.

Retries consume your budget. Every rejected request is effort spent on nothing, and it can push you across the threshold for a longer ban.

Escalation is one-directional. A site that was willing to throttle you is easy to convince that it should block you. Convincing it to reverse that decision is much harder.

Designing for backoff

The implementation points that matter:

• Exponential backoff with jitter, so a fleet of workers does not retry in unison.
• A global rate limiter, so total outbound requests have a ceiling regardless of individual worker behaviour.
• A circuit breaker, so a sustained failure rate pauses the pipeline rather than continuing to probe.
• A retry budget, so failures cannot consume more than a fixed share of requests before alerting.
• Separate handling per status, because a 429, a 403 and a 500 need different responses. The table in HTTP 409 Conflict covers the retry decision per code.

The pool-level machinery behind this is in What Is a Proxy Pool, and the concurrency side in Async Python Scraping Without Breaking Rate Limits.

Recognising silent throttling

Because no status code signals it, silent throttling hides. The clues are in the timing distribution: a rising median latency, or a bimodal distribution where a subset of requests hangs, suggests the server is delaying rather than refusing.

Set timeouts sensibly so you abandon rather than hang, and compare against a baseline measured earlier. The pattern is described in The 499 Status Code Explained, since delayed responses that you abandon produce that status in the server's logs.

When to stop entirely

If you have slowed down, honoured the guidance, used appropriate IPs, and the target still refuses, that is a considered answer rather than a puzzle. Some sites do not want automated collection, and their terms may say so.

The engineering instinct is to try harder. The correct conclusion is sometimes to stop, and it is worth stating plainly. The framing is in Is Web Scraping Legal and Data Collection Ethics for Engineering Teams.

Instrument the distinction

The four fields that make this diagnosis quick are the status code, whether a Retry-After header was present, the position in the run, and the exit IP. With those, the table above is answerable in seconds rather than by guesswork, and it is the foundation of Monitoring Scraper Health.