Using Python requests with Proxies

requests is where most Python scraping starts, and its proxy support is straightforward once you know the shape of the configuration. This is the practical set of patterns, including the instrumentation that makes failures diagnosable.

The basic configuration

A proxy is a dictionary keyed by scheme, or a single URL that applies to both.

import os
import requests

proxy = os.environ["LB_PROXY_URL"]  # http://user:pass@host:port

response = requests.get(
    "https://example.com",
    proxies={"http": proxy, "https": proxy},
    timeout=20,
)

The two keys matter because requests selects based on the destination scheme, not the proxy's. Setting only http means HTTPS destinations bypass the proxy silently, which is a common and confusing bug.

Prefer a Session

A Session reuses the underlying connection, which avoids repeating the TCP and TLS handshake on every call. For many small requests that overhead is a meaningful share of total time.

session = requests.Session()
session.proxies.update({"http": proxy, "https": proxy})
session.headers.update({"User-Agent": "Mozilla/5.0 (compatible; research-bot)"})

for url in urls:
    response = session.get(url, timeout=20)

Reuse the session, and set headers once rather than on every call.

Routing a single request through a different proxy

A per-request proxies argument overrides the session default. That is how you rotate without rebuilding the session.

response = session.get(url, proxies={"http": p, "https": p}, timeout=20)

Practical rotation logic uses a session value per unit of work rather than a new proxy per request, for the reasons in Rotating vs Sticky Proxies. A pool implementation with health tracking is in What Is a Proxy Pool.

Timeouts do two different things

The single timeout value sets both the connect and the read timeout. They fail for different reasons, and separating them makes diagnosis easier.

timeout = (10, 30)  # (connect, read)

A connect timeout suggests a network or proxy problem. A read timeout suggests a slow target or throttling. Treating them the same hides which one you are hitting, which matters because the remedies differ.

For mobile and residential proxies, set these generously. Variable path latency produces false failures on tight timeouts, as we explain in 4G, 5G, and LTE Mobile Proxies Explained.

Retries with backoff

Naive retry loops turn a soft rate limit into a hard block. Use a bounded retry with exponential backoff and jitter, and never retry a 4xx that indicates a permanent rejection.

from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

retry = Retry(
    total=3,
    backoff_factor=0.8,
    status_forcelist=[429, 500, 502, 503, 504],
    respect_retry_after_header=True,
    allowed_methods=["GET", "HEAD"],
)

adapter = HTTPAdapter(max_retries=retry)
session.mount("https://", adapter)
session.mount("http://", adapter)

respect_retry_after_header is the important setting: when a server tells you how long to wait, waiting is the fastest route back to success. We cover the interpretation in Rate Limiting vs Blocking.

Handling proxy authentication failures

A 407 means the proxy rejected you, not the target. Check for it explicitly, because retrying will never help.

response = session.get(url, timeout=(10, 30))
if response.status_code == 407:
    raise RuntimeError("Proxy authentication failed, check credentials")

The distinction between a 407 and a 403 is covered in Proxy Authentication, and it saves a lot of wasted debugging.

Log which exit IP served each request

This is the highest-value instrumentation you can add. Every failure becomes attributable rather than mysterious.

import logging

log = logging.getLogger(__name__)

def fetch(session, url, session_id):
    response = session.get(url, timeout=(10, 30))
    # A lightweight echo endpoint returns the address the target would see.
    exit_ip = requests.get(
        "https://api.ipify.org", proxies=session.proxies, timeout=10
    ).text
    log.info("url=%s status=%s exit_ip=%s session=%s", url, response.status_code, exit_ip, session_id)
    return response

In production, checking the echo endpoint on every request doubles your traffic. A better pattern is to verify the exit IP once per session, and log the session identifier with every request so the mapping is recoverable. The proxy checker gives you the exit IP and latency for verification, and IP lookup resolves it to a location.

Common mistakes

• Setting only the http key, causing HTTPS to bypass the proxy.
• Embedding credentials in source. Use environment variables, per Proxy Authentication.
• No timeout, so one hanging request stalls the job.
• Unbounded retries, which escalate rate limits into blocks.
• Ignoring Retry-After, which is free guidance the server is giving you.
• Not reusing a Session, paying handshake costs on every call.

Next steps

With a working client, the next problems are parsing accuracy and scale. Parsing HTML and JSON Reliably covers validation, and Async Python Scraping Without Breaking Rate Limits covers going concurrent without triggering defences.