Benchmarking Browser Automation Setups

Most automation benchmarks measure the wrong thing. They fetch a single easy page, report a latency figure, and conclude that one setup is faster. That tells you almost nothing about which one will finish your job.

A useful benchmark answers one question: what does it cost, in time and money, to get a correct result?

What to measure

Four numbers, in priority order.

Success rate. The fraction of attempts that return the data you wanted, validated against a schema rather than a status code. This dominates everything else, which is the point of Latency vs Success Rate.

Time per successful page. Wall-clock time divided by successes, not by attempts. A fast failure is not fast.

Cost per successful page. Proxy bandwidth plus compute plus any per-page service fee, divided by successes. This is the number that decides procurement.

Variance. Whether the results are stable run to run. A setup that succeeds 90 percent of the time on average but occasionally fails entirely is harder to operate than one that is consistently at 85.

Control the variables

A benchmark is only useful if you change one thing at a time.

Fix the target and the page set. Use your real pages, including the awkward ones: a search result, a detail page, a paginated listing. A benchmark on the homepage predicts nothing.

Fix the request set. Same URLs, same order if sequencing matters.

Fix the concurrency. Pushing harder degrades every setup differently, so hold it constant.

Fix the time window. Targets behave differently through the day, and a comparison run at different hours is not a comparison.

Vary one dimension. IP type, or client, or timeout. Not two at once, or you cannot attribute the result.

That last rule is where most benchmarks go wrong. Changing the browser and the proxy simultaneously produces a number with no interpretation.

The test matrix

A practical design with a manageable number of cells:

You do not need every combination. A useful reduction is to test the client and IP type across all page types, and treat concurrency as a separate tuning exercise once you know which client and IP you are using.

Sample size

Small samples lie. A run of twenty requests can show a 55 percent success rate that is really 80 percent over a hundred.

A few hundred attempts per cell gives a reasonably stable estimate without taking all day. If the difference between two options is inside the noise, they are the same for your purposes, and you should choose on cost.

Track the count alongside the rate so you can see whether a result is underpowered. That is the same discipline as any metric in Monitoring Scraper Health.

Measuring bandwidth correctly

Proxy cost depends on bytes transferred, so measure rather than estimate.

• Count both directions, though responses dominate.
• Include retries, since failures consume bandwidth and that is part of the cost case.
• Note whether assets were blocked, because a browser loading images can transfer an order of magnitude more than one that does not. The technique is in Playwright Scraping with Proxies.
• Record the exit IP so you can confirm the IP type you intended, using the proxy checker for verification.

A reusable harness

The shape that works:

import time
import statistics

def benchmark(fetch_fn, urls, endpoint):
    results = []
    for url in urls:
        started = time.monotonic()
        try:
            response = fetch_fn(url, endpoint)
            ok = validate(response)          # schema check, not status code
            bytes_in = len(response.content)
        except Exception:
            ok, bytes_in = False, 0
        results.append({
            "url": url,
            "ok": ok,
            "duration": time.monotonic() - started,
            "bytes": bytes_in,
        })
    successes = [r for r in results if r["ok"]]
    return {
        "success_rate": len(successes) / len(results),
        "seconds_per_success": sum(r["duration"] for r in results) / max(1, len(successes)),
        "bytes_per_success": sum(r["bytes"] for r in results) / max(1, len(successes)),
        "p50": statistics.median([r["duration"] for r in successes] or [0]),
    }

Two details carry most of the value. The validate call is what makes success meaningful, since a challenge page loads successfully. And dividing by successes rather than attempts is what separates a cheap setup from a fast one.

Reporting the result

A comparison worth acting on has this shape:

The interpretation is not "pick the highest success rate". It is: if the residential HTTP client reaches your required reliability, use it and keep the difference. If your target demands 95 percent, the browser earns its cost.

The numbers above are illustrative rather than measured, and should be replaced with your own. What matters is the shape of the decision.

What to do with the answer

Two outcomes are common.

A cheap setup is sufficient. Then stop optimising and put your effort into parsing accuracy and monitoring. The pipeline, not the transport, is where the remaining gains are, per Parsing HTML and JSON Reliably.

Only an expensive setup works. Then decide whether the target justifies the cost. Sometimes the honest answer is that a particular source is too defended to collect economically, and the effort belongs elsewhere. That is a legitimate conclusion, discussed in Anti-Scraping Techniques and How to Respond.

Re-run it

A benchmark is a snapshot, not a fact. Targets change their defences, providers change their pools, and your code changes. Re-running the matrix periodically turns a one-off decision into a maintained understanding, and it catches degradation before it affects production.

For the tooling comparison the matrix is testing, see Browser Automation Tools Compared, and for the network side, Choosing a Proxy for Web Scraping.