Configuring Scrapy Proxy Middleware

Scrapy is a framework rather than a script, which means it already has the concepts a resilient scraper needs: middleware, throttling, retry policy and concurrency limits. Wiring a proxy into those correctly gives you behaviour that individual scripts recreate badly.

The simplest option

Scrapy ships with an HTTP proxy middleware. Enable it and set the meta key per request.

# settings.py
DOWNLOADER_MIDDLEWARES = {
    "scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware": 110,
}

yield scrapy.Request(url, meta={"proxy": "http://user:pass@host:port"})

That is enough for a fixed endpoint. It does nothing useful for rotation, which is what most projects actually need.

A rotating middleware

A custom middleware picks an endpoint per request and records the outcome so failing endpoints are avoided.

import random

class RotatingProxyMiddleware:
    def __init__(self, endpoints, max_failures=3, cooldown=60):
        self.endpoints = [
            {"url": url, "failures": 0, "cooldown_until": 0.0} for url in endpoints
        ]
        self.max_failures = max_failures
        self.cooldown = cooldown

    @classmethod
    def from_crawler(cls, crawler):
        return cls(crawler.settings.getlist("PROXY_ENDPOINTS"))

    def _pick(self):
        import time
        now = time.monotonic()
        available = [e for e in self.endpoints if e["cooldown_until"] <= now]
        return random.choice(available or self.endpoints)

    def process_request(self, request, spider):
        endpoint = self._pick()
        request.meta["proxy"] = endpoint["url"]
        request.meta["proxy_endpoint"] = endpoint

    def process_response(self, request, response, spider):
        endpoint = request.meta.get("proxy_endpoint")
        if endpoint is None:
            return response
        if response.status in (403, 429):
            self._penalise(endpoint)
        else:
            endpoint["failures"] = 0
        return response

    def process_exception(self, request, exception, spider):
        endpoint = request.meta.get("proxy_endpoint")
        if endpoint is not None:
            self._penalise(endpoint)

    def _penalise(self, endpoint):
        import time
        endpoint["failures"] += 1
        if endpoint["failures"] >= self.max_failures:
            endpoint["cooldown_until"] = time.monotonic() + self.cooldown
            endpoint["failures"] = 0

Register it and provide the endpoints:

DOWNLOADER_MIDDLEWARES = {
    "myproject.middlewares.RotatingProxyMiddleware": 110,
}

PROXY_ENDPOINTS = [
    "http://user:pass@host:port-session-a",
    "http://user:pass@host:port-session-b",
]

Notice that each endpoint carries its own session value, which is how stickiness is expressed. The mechanics are in Understanding Proxy Session IDs.

Separating transport failures from target rejections

The middleware above penalises both connection errors and 403/429 responses. In production these deserve different handling.

A connection error is an endpoint problem. A 403 is the target reacting to your request pattern or the address's reputation, and a 429 is an explicit rate limit that should be honoured rather than rotated around.

def process_response(self, request, response, spider):
    if response.status == 429:
        retry_after = response.headers.get("Retry-After")
        # Back off globally rather than blaming one endpoint.
        spider.crawler.engine.pause()
        # ... schedule resume
    return response

The distinction is the one we draw in Rate Limiting vs Blocking, and getting it wrong sends you rotating away from working endpoints for no reason.

Throttling settings that matter

Scrapy's defaults are polite. Tighten or loosen them deliberately rather than turning them off.

AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_START_DELAY = 1.0
AUTOTHROTTLE_MAX_DELAY = 30.0
AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0   # per domain

DOWNLOAD_DELAY = 0.5
RANDOMIZE_DOWNLOAD_DELAY = True          # jitter is not optional

AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0 is the important correction. Scrapy's default target is higher, but for defended targets one request at a time per domain with real pacing is more likely to succeed than four in parallel. The reasoning is in How Anti-Bot Systems Detect Scrapers.

Retry configuration

Scrapy retries certain status codes by default. Tune the list and cap the attempts.

RETRY_ENABLED = True
RETRY_TIMES = 2
RETRY_HTTP_CODES = [429, 500, 502, 503, 504, 522, 524]

Keep RETRY_TIMES low. Unbounded retries are a common way to escalate a soft limit into a hard block, and the middleware should move to a fresh endpoint rather than hammering one.

Logging the exit IP

Attribution is what turns a mystery into a fix. Log the endpoint or session with each response so a failure can be traced to an address.

def process_response(self, request, response, spider):
    spider.logger.debug(
        "response url=%s status=%s session=%s",
        response.url,
        response.status,
        request.meta.get("proxy_endpoint", {}).get("url"),
    )
    ...

Verify an endpoint independently with the proxy checker, which reports the exit IP and added latency, and confirm location with IP lookup if you are using targeting.

Where to go next

With middleware in place, the remaining work is parsing accuracy and health monitoring. Parsing HTML and JSON Reliably covers validation, and Monitoring Scraper Health covers the metrics that tell you the spider is degrading before it fails. For the general proxy layer, What Is a Proxy Pool describes the design this middleware implements.