Anti-Scraping Techniques and How to Respond

Sites defend themselves in a small number of recognisable ways. Knowing which technique you are facing tells you whether to slow down, change your client, or conclude that collection is not welcome here.

That last outcome is a legitimate result, not a failure.

Rate limiting

The gentlest defence. The server accepts your requests but refuses or delays them past a threshold, usually returning 429 and often a Retry-After header.

What it signals: the site objects to your volume, not necessarily to your existence. Search and API endpoints are commonly rate limited for everyone.

The response: slow down, honour Retry-After, reduce concurrency, and spread requests across more addresses if the limit is per IP. Never retry immediately, which escalates a warning into a block. The details are in Rate Limiting vs Blocking.

WAF and IP-level blocking

A web application firewall or edge service refuses the request outright, often with a 403, based on the IP's reputation or ASN.

What it signals: your network identity was classified before the request reached the application.

The response: change IP type. Datacenter ranges are frequently blocked wholesale, which is the situation described in Datacenter Proxies: Speed vs Detectability. If residential addresses are also refused, the site may be rejecting automation categorically rather than by type.

JavaScript and browser challenges

The server returns an interstitial that must be solved by executing JavaScript before the real content is served. Cloudflare's checks and similar systems are the common examples.

What it signals: the site wants to verify you are a browser rather than an HTTP client.

The response: use a real browser engine, which handles these natively in most cases. Playwright and Selenium both work here, compared in Playwright vs Selenium. A solver service is a workaround rather than a strategy, covered in Working Through CAPTCHAs in Automation.

CAPTCHAs

A visible challenge, sometimes interactive, triggered by a risk score rather than by volume alone.

What it signals: your combined signals crossed a threshold, commonly because the IP is low-trust.

The response: reduce the score instead of paying per solve. Better IPs lower challenge frequency dramatically, which is the argument in Why Good Proxies Reduce CAPTCHA Frequency. Solver services have a role for occasional cases, but they are a recurring cost and their accuracy and terms are worth checking.

Device and browser fingerprinting

The site collects characteristics such as canvas output, WebGL renderer, fonts and screen metrics, then looks for patterns across sessions.

What it signals: the site is matching clients rather than just addresses.

The response: use a consistent, realistic fingerprint. If you are running many identities, an antidetect browser is the appropriate tool, covered in Antidetect Browsers Compared. The common error is a realistic network with an inconsistent client, which we describe in Why Antidetect Browsers Need Proxies.

Honeypots

Decoy content that a human would never interact with: hidden links, fields invisible to a browser, or elements positioned off-screen. Following them marks you as automation.

What it signals: nothing about you in particular. It is a trap, not a response.

The response: only interact with elements a human could perceive and act on. Skip hidden links, never fill hidden form fields, and click targets in positions a person would see. This is a reason to be cautious with aggressive recursive crawling, since following every link is exactly what honeypots exploit.

Tarpits

The server responds deliberately slowly, sometimes for minutes, rather than blocking. The intent is to waste your resources and make collection uneconomic.

What it signals: the site has decided to impose a cost rather than refuse.

The response: set sane timeouts so you abandon rather than hang, and treat persistent slowness as a signal to reduce volume or reconsider the target. Do not increase timeouts to push through it, since that is exactly what the defence is designed to exploit.

Behavioural analysis

The site watches timing, navigation sequences, mouse and scroll events, and interaction patterns to distinguish people from scripts.

What it signals: the site values behaviour over static signals.

The response: behave more plausibly. Vary timing, follow sensible navigation paths rather than jumping directly to deep URLs, and avoid exact repetition. The nuance is in Human-Like Behaviour in Automation.

Content fingerprinting and canaries

Some defences insert unique markers per response so they can identify which scraper is reusing content, or which path an automated client followed.

What it signals: they care about downstream use as well as collection.

The response: respect it. This is usually a sign that republishing the data is unwelcome, which is a terms issue rather than a technical one.

The honest decision framework

For each defence, you have three choices, and only two are technical:

1. Adapt technically, when the defence is a generic protection and your collection is permitted.
2. Reduce scope or volume, when the defence signals that your current approach is unwelcome but some access is acceptable.
3. Stop, when the site has clearly refused access or the terms prohibit what you are doing, or when the data is behind authentication you do not hold.

Choosing option three is not a defeat. A site that answers every request with a challenge, a tarpit or an explicit refusal has told you something, and the engineering answer is not always to escalate. The legal and ethical framing is in Is Web Scraping Legal and Data Collection Ethics for Engineering Teams.

Diagnosing which one you face

Record enough per request to tell the cases apart: status code, response length, time to first byte, and whether the body contains the data you expected. Those four fields distinguish a rate limit from a block, a challenge from a tarpit, and a successful fetch from a silently empty one. That instrumentation is the foundation of Monitoring Scraper Health.