Fixing 403 Forbidden Errors When Scraping

A 403 means the server understood your request and refused it. That is all it means. The cause could be your proxy, your headers, your cookies, your client fingerprint, your IP reputation or the resource itself, and the list is long enough that guessing wastes hours.

Work through it in order instead.

First: is it a proxy problem or a target problem?

This distinction saves the most time, so make it first.

• 407 Proxy Authentication Required is the proxy rejecting you. Credentials, format or allowlisting. We cover it in Proxy Authentication.
• 403 Forbidden is the target rejecting you. The proxy accepted the connection, and the origin or its edge decided against you.

If you see a 403, stop looking at proxy configuration. The tunnel worked.

Reproduce it outside your code

Before changing anything, confirm the behaviour with curl. That separates a code problem from an environment problem.

curl -v -x "http://lb-USER:SECRET@proxy.lightningbytes.com:1080" \
  -A "Mozilla/5.0 (compatible; research-bot)" \
  -H "Accept: text/html,application/xhtml+xml" \
  -o /dev/null -w "%{http_code}\n" \
  https://example.com/page

If curl succeeds and your code fails, the difference is in your client. If both fail, work through the list below.

Also test the same URL without the proxy. If the direct request succeeds and the proxied one fails, the IP is the variable. That is the single most informative test in this whole process.

Step 1: IP reputation and type

The most common cause on defended sites. Hosting ASNs are classified, and a 403 arrives before the application even looks at your headers.

Check what address you are actually exiting from. The proxy checker reports the exit IP and added latency, and IP lookup shows the network and country.

If you are on a datacenter address and the target defends itself, that is the answer, and the remedy is a different IP type rather than more header tuning. See Datacenter Proxies: Speed vs Detectability.

Step 2: User agent and headers

Some sites reject specific user agents outright, and others reject requests whose header set does not match the claimed client.

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0 Safari/537.36",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    "Accept-Language": "en-US,en;q=0.9",
    "Accept-Encoding": "gzip, deflate, br",
    "Connection": "keep-alive",
}

Two cautions. First, if you claim to be Chrome, send the headers Chrome sends. A subset is itself a signal, as described in How Anti-Bot Systems Detect Scrapers. Second, check for a missing Accept-Language, which many scripts omit and browsers always send.

Step 3: Cookies

Some sites require a session cookie that is set on the first visit. A client that discards cookies will be refused on every subsequent request.

session = requests.Session()  # persists cookies across requests
session.headers.update(headers)
session.get("https://example.com/")       # acquire cookies
response = session.get("https://example.com/page")

If you are rotating IPs, note that a cookie issued to one session may not be valid from another, which is the coherence problem in Rotating vs Sticky Proxies.

Step 4: Referer

A surprising number of sites check Referer for navigation that should have come from within the site. Sending the expected value resolves a whole class of 403s.

session.headers["Referer"] = "https://example.com/"

Do not fabricate an inconsistent referer chain, but do send the one a real navigation would produce.

Step 5: Client fingerprint

If headers and cookies are correct and the 403 persists, the site may be fingerprinting your client at the TLS or HTTP/2 level. A Python client does not look like Chrome regardless of its headers. This is covered in TLS Fingerprinting and Scraping.

The remedy is to use a real browser engine, which brings a consistent fingerprint. The options are in Playwright vs Selenium.

Step 6: Rate and behaviour

A 403 that appears after a number of successful requests is usually behavioural. Look for a preceding 429 in your logs, since ignoring one often produces a 403 shortly afterwards, as we describe in Rate Limiting vs Blocking.

Slow down, reduce concurrency per address, and add jitter. Machine-regular timing is a signature in its own right.

Step 7: The resource itself

Not every 403 is about you. Some resources are genuinely restricted: a page available only to logged-in users, a region-limited asset, or an endpoint that has been disabled.

Test with a direct request from an ordinary browser. If a human cannot access it either, the 403 is correct and the data is not available to you.

A decision tree

407?                    -> proxy credentials or allowlisting
403 on proxied request
  and 200 direct?       -> IP reputation, change IP type
403 on both?            -> headers, cookies, referer, fingerprint
403 only after N ok?    -> rate limiting or behaviour
403 on a URL a browser
  also cannot open?     -> the resource is restricted, not your problem

When to stop

If you have worked through the list and the site continues to refuse, consider that the refusal may be the intended answer. A site with an explicit anti-automation policy, or one that requires authentication you do not hold, is not a configuration problem.

The boundary is set by the site's terms and by law, not by whether you can find a working combination. We cover the framing in Is Web Scraping Legal and the practical limits in Anti-Scraping Techniques and How to Respond.

Instrument so you do not repeat this

Log the status code, the exit IP and the response length for every request. With those three fields, the next 403 takes minutes to diagnose rather than hours, because you can immediately see whether it correlates with a specific address or followed a rate-limit response. The setup is described in Monitoring Scraper Health.