Fixing 503 Service Unavailable

A 503 says the server cannot handle the request right now. That could mean the site is down, overloaded, or deliberately refusing you, and the three cases call for very different responses.

Getting the diagnosis right matters because the wrong response makes the situation worse.

The three causes

Genuine unavailability. Maintenance, a crash, a deploy, or an upstream dependency failing. Nothing to do with you, and it affects everyone.

Overload. The server is shedding load to protect itself. Common during traffic spikes, and increasingly common for sites under sustained scraping pressure.

Deliberate soft blocking. Some defences return 503 rather than 403, particularly when they want to discourage without an explicit refusal. This is a signalling choice, and it is easy to mistake for an outage.

The signals that distinguish them

Check without the proxy. If a direct request also fails, the site is down or overloaded for everyone.

Check the body. A maintenance page, a branded error page or a CDN's default interstitial all mean something specific. A bare 503 with no body is more likely to be a soft block.

Check the headers. A Retry-After header suggests a managed, intentional response rather than a crash. A Server header naming a CDN or WAF points to the edge layer rather than the application.

Check the pattern. If some IPs fail and others succeed, it is per-identity rather than global.

Check other endpoints. If the homepage works and one path does not, it is not a site-wide outage.

Honour Retry-After when it is present

When the server tells you how long to wait, that is the fastest route back to success. Ignoring it is the most reliable way to turn a temporary refusal into a longer one.

import time

def retry_after_seconds(response, default=30):
    value = response.headers.get("Retry-After")
    if value is None:
        return default
    if value.isdigit():
        return int(value)
    # It can also be an HTTP date.
    try:
        from email.utils import parsedate_to_datetime
        delta = parsedate_to_datetime(value) - datetime.now(timezone.utc)
        return max(1, int(delta.total_seconds()))
    except Exception:
        return default

Two formats are permitted: a number of seconds, or an HTTP date. Handling both avoids the common bug where a date-valued header is ignored and a default is applied instead.

Backoff that actually recovers

Exponential backoff with jitter is the standard, and the jitter is not optional. Without it, every worker that failed at the same moment retries at the same moment, which recreates the spike that caused the problem.

import random
import time

def backoff(attempt, base=1.0, cap=60.0):
    delay = min(cap, base * (2 ** attempt))
    return random.uniform(0, delay)

Cap the delay so a long outage does not push a retry hours into the future, and cap the attempts so a permanently unavailable resource does not consume a worker indefinitely.

For a fleet of workers, consider a shared circuit breaker: when the error rate crosses a threshold, stop sending for a cooldown rather than letting each worker discover the problem independently. That pattern is described in What Is a Proxy Pool.

The retry storm

This is the failure mode to avoid. A site returns 503 under load, your workers retry immediately, the additional load extends the outage, and the cycle continues. Everyone involved is worse off, including you.

Three guards:

• Global rate limiter, so total outbound requests cannot exceed a ceiling regardless of how many workers are retrying.
• Circuit breaker, so a sustained failure rate pauses the whole pipeline.
• Retry budget, so failures cannot consume more than a fixed share of your requests. Once exceeded, the pipeline alerts rather than retries.

The general principle is in Rate Limiting vs Blocking, and it is worth reading alongside this post.

If it is a soft block

When the evidence points to deliberate blocking rather than an outage, the response is different:

• Reduce volume. Fewer requests, lower concurrency and longer spacing.
• Check your IP type. If you are on datacenter addresses, that may be the whole story, per Datacenter Proxies: Speed vs Detectability.
• Verify your exit location with IP lookup if the site varies by region.
• Consider whether collection is welcome. A site returning 503 consistently, to every IP type, is telling you something. That is a legitimate signal to stop, as discussed in Anti-Scraping Techniques and How to Respond.

A practical checklist

1. Does it fail without the proxy too?     -> site outage, wait
2. Is there a Retry-After header?          -> honour it exactly
3. Is it one endpoint or the whole site?   -> endpoint-specific problem
4. Do some IPs succeed?                    -> identity-specific, change IP type
5. Does it follow a rate-limit response?   -> soft block, slow down
6. Is the error rate sustained?            -> circuit break, alert, do not hammer

Instrument for the diagnosis

Record status code, exit IP, presence of Retry-After, and whether a direct request also fails. Those four fields turn a 503 from a mystery into a diagnosis in under a minute, and they belong in the monitoring described in Monitoring Scraper Health.

For neighbouring cases, The 499 Status Code Explained covers the status that means your own client gave up, and Fixing 403 Forbidden Errors When Scraping covers the explicit refusal.