Playwright Scraping with Proxies

Playwright's design makes it a good fit for scraping at modest scale. Browser contexts are isolated by default, proxies can be set per context rather than per browser, and request interception is built in rather than bolted on.

That combination means one browser process can serve several distinct identities cleanly.

Configuring a proxy

Two levels are available: the browser launch and the context.

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(
        proxy={"server": "http://proxy.lightningbytes.com:1080"}
    )
    page = browser.new_page()
    page.goto("https://api.ipify.org")
    print(page.inner_text("body"))
    browser.close()

Credentials go in the same dictionary:

proxy = {
    "server": "http://proxy.lightningbytes.com:1080",
    "username": "lb-USERNAME",
    "password": "SECRET",
}

The advantage over Selenium is that Playwright handles proxy authentication natively, so no extension or local relay is required. That removes one of the awkward parts of browser automation we describe in Selenium Proxy Setup and Rotation.

Parallel identities with contexts

browser.new_context() creates an isolated session with its own cookies and storage. Because proxies can be set per context, one browser process can run several identities in parallel.

endpoints = [
    {"server": "http://proxy.lightningbytes.com:1080",
     "username": "lb-USER-session-a", "password": "SECRET"},
    {"server": "http://proxy.lightningbytes.com:1080",
     "username": "lb-USER-session-b", "password": "SECRET"},
]

contexts = [browser.new_context(proxy=e) for e in endpoints]
try:
    for context in contexts:
        page = context.new_page()
        page.goto("https://example.com")
finally:
    for context in contexts:
        context.close()

The session value in each username is what gives each context a distinct exit address. Without it, both contexts would share one IP and you would have halved your pool without noticing, which is the mistake described in Understanding Proxy Session IDs.

Keep the number of concurrent contexts modest. Each is a real browser page, and both your machine and the target have limits. The general pacing argument is in Rate Limiting vs Blocking.

Cutting bandwidth with request interception

Rendered pages pull in far more bytes than the data you want. Blocking non-essential resources is often the single largest cost reduction available.

BLOCKED = {"image", "media", "font", "stylesheet"}

context = browser.new_context(proxy=proxy)
context.route(
    "**/*",
    lambda route: route.abort()
    if route.request.resource_type in BLOCKED
    else route.continue_(),
)

Two cautions. Blocking stylesheets can change layout-dependent selectors, so verify your extraction still works. And some sites detect missing resource requests as an automation signal, so this is a trade-off rather than a free win.

The bandwidth case is made in What Is Proxy Bandwidth, and the general approach to cutting consumption is in How Much Residential Bandwidth Do You Need.

Waiting for the right thing

The most common source of flaky browser scraping is waiting for the wrong signal. Prefer waiting for a specific element or network event over a fixed sleep.

page.goto("https://example.com/products", wait_until="domcontentloaded")
page.wait_for_selector("[data-testid='product-card']", timeout=20_000)

If the data arrives via an XHR, waiting for that response is even more precise, and extracting from the response body is lighter than parsing the DOM. The technique is in Stop Scraping the Page, Find the API Instead.

Consistency: the part that decides outcomes

A real browser gives you a realistic fingerprint, which is a genuine advantage over an HTTP client. It does not guarantee consistency between that fingerprint and the rest of your session.

Check that these agree:

Timezone and locale with the proxy's location.
User agent and viewport with each other, particularly if you override either.
Headers the browser sends versus what you added manually.

A desktop user agent from a mobile-emulated viewport, or a US address with a European timezone, are mismatches that detection systems test for directly. We cover the logic in Why Antidetect Browsers Need Proxies.

Verifying the proxy is actually in use

Silent misconfiguration is common, and a request that succeeds while bypassing the proxy looks identical to success. Verify the exit IP at the start of each session.

The proxy checker confirms the exit IP and added latency for the endpoint, and the WebRTC leak test catches browser channels that expose your real address around the proxy, which is a different failure from a misconfigured proxy.

Where Playwright fits

Playwright is the stronger default for new browser automation work: better waiting semantics, native proxy authentication and clean context isolation. Selenium remains a reasonable choice for existing projects and wider language support, compared in Playwright vs Selenium.

If the target does not require JavaScript execution, an HTTP client is faster and lighter. Beautiful Soup: Parsing HTML Without the Headache covers the lighter path.