Async Python Scraping Without Breaking Rate Limits

Async Python makes concurrent requests easy, which is exactly why it causes so many problems. It is trivial to fire a thousand requests at a target, discover that throughput collapses under throttling, and conclude that your proxies are bad.

Concurrency is a dial, not a switch. This is how to set it properly.

Why more concurrency often means less throughput

A target that rate limits responds to aggression by slowing down, returning 429s, or introducing challenges. Each of those reduces your effective throughput. Past a certain point, adding concurrency adds failures and retries without adding results.

The counterintuitive lesson: the optimal concurrency is usually lower than your machine allows, and it is set by the target and your proxies rather than by your CPU.

The measurement that matters is successful pages per minute, not requests per second. That is the same argument as Latency vs Success Rate, applied to concurrency.

A bounded async fetcher

The pattern is a semaphore per host, plus one request at a time per proxy endpoint.

import asyncio
import httpx

class Fetcher:
    def __init__(self, endpoints, per_host=2):
        self.endpoints = endpoints
        self.per_host = per_host
        self.host_sems: dict[str, asyncio.Semaphore] = {}
        self.endpoint_locks = {e["url"]: asyncio.Lock() for e in endpoints}

    def _host_sem(self, host: str) -> asyncio.Semaphore:
        if host not in self.host_sems:
            self.host_sems[host] = asyncio.Semaphore(self.per_host)
        return self.host_sems[host]

    async def fetch(self, client, url, endpoint):
        host = httpx.URL(url).host
        async with self._host_sem(host):
            async with self.endpoint_locks[endpoint["url"]]:
                return await client.get(url, timeout=(10, 30))

Two semaphores do different jobs. The host semaphore caps pressure on the target. The endpoint lock enforces one request at a time per address, which is what stops you from defeating your own pool by parallelising onto one IP, the mistake described in What Is a Proxy Pool.

httpx versus aiohttp

Both are capable. httpx has a requests-like API and clean proxy support, which makes it easier to port existing code.

async with httpx.AsyncClient(proxy="http://user:pass@host:port") as client:
    response = await client.get("https://example.com", timeout=(10, 30))

aiohttp is leaner and has a longer history in scraping. Either works; pick based on familiarity rather than benchmarks.

Throttling and backoff

Concurrency limits alone are not enough. You also want spacing between requests, jittered so it does not look mechanical.

import random

async def polite_fetch(client, url, endpoint, min_delay=0.8):
    await asyncio.sleep(random.uniform(min_delay, min_delay * 2))
    return await client.get(url, timeout=(10, 30))

And treat 429 properly: back off, honour Retry-After, and slow the pool rather than the individual request.

if response.status_code == 429:
    retry_after = response.headers.get("Retry-After")
    delay = float(retry_after) if retry_after and retry_after.isdigit() else 30
    await asyncio.sleep(delay)

A blanket sleep is crude but effective. A better version reduces the host semaphore's effective rate for a window. Either is far better than retrying immediately, which escalates a soft limit into a block, as described in Rate Limiting vs Blocking.

Session stickiness in async code

Rotation interacts awkwardly with concurrency. If two tasks share a session, they share an address and you lose the parallelism you intended.

Give each task its own endpoint or session value, and keep the mapping explicit. The mechanism is in Understanding Proxy Session IDs, and the pool design in What Is a Proxy Pool.

For a multi-step flow, do not run the steps concurrently. Sequence them on one session so the identity stays coherent, which is the argument in Rotating vs Sticky Proxies.

Measuring what you actually got

Instrument the run rather than trusting that concurrency equals speed:

• Successful results per minute, the metric that matters.
• Status code distribution, with 429s broken out separately.
• P50 and P95 latency, since averages hide the tail that governs your timeouts.
• Exit IP per request, so a proxy problem is distinguishable from a target problem.

Those metrics are the basis of Monitoring Scraper Health.

A practical starting point

Set host concurrency to two, one request per endpoint, and a jittered delay of about a second. Run a few hundred requests and measure successful pages per minute. Then adjust one variable at a time: raise host concurrency, or lower the delay, and see whether throughput improves or the failure rate rises.

For most defended targets, you will find the optimum quickly and it will be lower than you expected. The same tuning applies to the patterns in Using Python requests with Proxies and Scrapy Proxy Middleware.