How Anti-Bot Systems Detect Scrapers

Blocking is not a switch. Modern bot management assigns a risk score to each request and acts on thresholds, and it weighs several independent signals. Understanding the layers explains why a change that should have fixed everything sometimes changes nothing.

The layers, in the order they are usually applied

Defenders apply cheap checks first, because they run on every request.

1. Network identity. The ASN the request came from, and the address's history. Hosting ranges are classified almost instantly, which is why this is the single largest factor and the reason for Datacenter Proxies: Speed vs Detectability.

2. Request metadata. Headers, including user agent, Accept variants, header order and the presence or absence of the ones real browsers always send. A Python client is missing several headers that a browser includes.

3. TLS and HTTP/2 fingerprints. The client's cryptographic and protocol behaviour, which identifies the software stack independently of what the headers claim. This is covered in TLS Fingerprinting and Scraping.

4. Session coherence. Whether the IP, timezone, language, cookies and any account are consistent with each other. An address in one country with a timezone from another is a direct contradiction.

5. Behaviour. Request rate, timing distribution, navigation sequence, and whether the session reads like a person or a loop.

6. Challenge response. Whether the client can execute JavaScript and answer a challenge correctly, which distinguishes a browser from a script.

Why fixing one layer rarely works

The score is cumulative. Clearing the network gate by using a residential IP removes one input, but if the TLS fingerprint says Python, the headers say Python and the behaviour says loop, the score stays high.

Equally, correcting the client while using a classified datacenter IP produces an immediate block at layer one, before any of the other work is evaluated.

The practical implication is that you should fix the layer that is currently binding, and identify it by experiment rather than assumption. Hold the client constant and change the IP type, then hold the IP type constant and change the client. Whichever flips the outcome is your bottleneck.

The signals that most often give people away

In practice, these account for the majority of avoidable blocks:

• Machine-regular timing. Identical intervals between requests do not occur naturally.
• Concurrency on one address. Several parallel requests from one IP is not how a person browses.
• Mismatched headers. A claimed browser with a header set that browser never sends.
• Missing headers. Real browsers send a predictable list; scripts often send a subset.
• Ignoring rate-limit responses. Continuing after a 429 turns a warning into a block, per Rate Limiting vs Blocking.
• Inconsistent geography. Timezone, language and IP disagreeing.
• No cookie continuity. A client that discards cookies never looks like a returning visitor.

Matching your client to your claim

The clearest principle: if you claim to be a browser, behave like one. Either use a real browser engine, which brings a consistent fingerprint, or identify honestly and accept the higher scrutiny that comes with it. The hybrid, where a library pretends to be Chrome, is the worst of both, because it is a detectable contradiction.

That is the core argument in Why Antidetect Browsers Need Proxies, applied to automation generally.

What actually reduces the score

A practical checklist, in rough order of effect:

1. Use an IP type the target accepts. Residential or mobile for defended consumer sites.
2. Run a real browser if the site executes JavaScript or fingerprints clients.
3. Pace requests with jitter, and honour Retry-After.
4. Keep concurrency at one per address, adding addresses rather than parallelism.
5. Make geography consistent, verifying the exit location with IP lookup.
6. Maintain cookies within a session so you look like a returning visitor.
7. Rotate between units of work, not within them, per Rotating vs Sticky Proxies.
8. Retire addresses that fail, rather than retrying them into the ground.

The measurement that settles arguments

Rather than reasoning about which layer is at fault, measure. Run the same request set with controlled variations and record acceptance rate. That is the method in Latency vs Success Rate, and it converts a debate into a number.

The proxy checker gives you the exit IP and added latency, which rules the network layer in or out quickly. If a clean endpoint still gets challenged, the problem is above the network.

The limits of the exercise

Everything above is about not being misclassified when you are doing something permitted. It is not a method for defeating access controls you have been denied.

Where a site has explicitly refused access, or where the data is behind authentication you do not hold, better infrastructure does not create permission. The reasoning is in Is Web Scraping Legal and Data Collection Ethics for Engineering Teams.

Within those bounds, the goal is unremarkable: look like the visitor the site expects, so it serves you what it serves everyone else.