Using FlareSolverr with Proxies

FlareSolverr is a small service that solves certain browser challenges on your behalf and returns the resulting page. It exists because a class of sites serves an interstitial that a plain HTTP client cannot get past.

It works, within limits. Understanding those limits prevents a lot of wasted time.

What it does

FlareSolverr runs a headless browser behind an HTTP API. You send it a URL, it loads the page in a real browser, waits for any challenge to complete, and returns the resolved HTML along with the cookies it collected.

curl -X POST http://localhost:8191/v1 \
  -H "Content-Type: application/json" \
  -d '{"cmd": "request.get", "url": "https://example.com/page", "maxTimeout": 60000}'

The response contains the page content and the cookies needed for subsequent requests through the same session. That is the useful part: once solved, you can continue with a plain HTTP client using those cookies, rather than rendering every page.

Why it is needed

Some sites, particularly those fronted by Cloudflare's challenge pages, return an interstitial that requires JavaScript execution and a period of waiting before the real content is served. An HTTP client receives only the interstitial, and its challenge tokens cannot be generated without a browser.

FlareSolverr fills that gap by being the browser. The background is in Anti-Scraping Techniques and How to Respond.

Configuring a proxy

The service accepts a proxy, which matters because the challenge is frequently triggered by the IP in the first place.

{
  "cmd": "request.get",
  "url": "https://example.com/page",
  "maxTimeout": 60000,
  "proxy": {
    "url": "http://proxy.lightningbytes.com:1080",
    "username": "lb-USERNAME",
    "password": "SECRET"
  }
}

This is the part most people get wrong. Running FlareSolverr without a trusted proxy means it solves challenges from a datacenter address, which is exactly the condition that produces challenges. With residential or mobile IPs, many interstitials do not appear in the first place, so the solver is often unnecessary. That is the argument in Captchas and Proxies.

Wiring it into a scraper

The efficient pattern avoids using the solver for every page.

import requests

SOLVER = "http://localhost:8191/v1"

def solve(url, proxy):
    payload = {
        "cmd": "request.get",
        "url": url,
        "maxTimeout": 60000,
        "proxy": {"url": proxy["server"], "username": proxy.get("username"), "password": proxy.get("password")},
    }
    response = requests.post(SOLVER, json=payload, timeout=90)
    data = response.json()
    return data["solution"]["response"], data["solution"]["cookies"]

def fetch_rest(url, cookies, proxy_url):
    session = requests.Session()
    session.proxies = {"http": proxy_url, "https": proxy_url}
    for cookie in cookies:
        session.cookies.set(cookie["name"], cookie["value"], domain=cookie.get("domain"))
    return session.get(url, timeout=(10, 30))

Solve once, reuse the cookies, and fetch subsequent pages directly. That turns a per-page browser cost into a per-session one, which is the difference between viable and unviable at any real volume.

The real limitations

Be clear-eyed about these before designing around it.

It is reactive. Challenge systems change, and the solver lags. A working setup can break without your code changing.

It is slow. Each solve involves a real page load and a wait, typically several seconds. That cost multiplies if you solve per page.

It does not fix your signals. If your IP type, fingerprint or behaviour is the problem, the solver bypasses the symptom for that request while leaving the cause in place.

Detection exists for it. Some systems identify automated browsers, and a solver running headless may itself be flagged.

Memory and stability. Long-running browser processes leak and occasionally hang. Operationally, treat it as a service that needs supervision and periodic restarts, similar to the lifecycle concerns in Puppeteer Web Scraping.

Its use has a boundary. Circumventing a protection on a site that has refused you access is not made acceptable by a third-party tool. The framing is in Is Web Scraping Legal.

When it is the right tool

Three cases where it earns its place:

A site with a light, stable challenge where the solve succeeds reliably and the content is public. The cost per solve is acceptable at modest volume.

A one-time session establishment. Solving once to obtain cookies, then fetching directly for the rest of the run. This is where the tool is genuinely useful.

Legacy pipelines where replacing the component is not worth the effort, and the setup works.

When it is not

High volume. Per-page solving does not scale economically. Use a trusted IP type and a real browser integration instead, per Playwright Scraping with Proxies.

Frequent challenge changes. If your solves fail regularly, the maintenance burden exceeds the benefit.

As a first resort. Check the IP type and the client before adding a solver. Most frequent challenges are caused by a datacenter address, and moving to residential resolves them outright.

A better default

For most projects the sequence that works is: correct IP type, real browser where needed, sane pacing, then a solver only if a specific challenge remains. Confirm the network layer with the proxy checker so you know whether the challenge is about the address or the client, and track challenge rate over time per Monitoring Scraper Health so you can see whether the solver is still earning its place.