The 499 Status Code Explained
499 is non-standard and means the client closed the connection before the server responded. How to tell an aggressive timeout from server throttling.
- tutorials
- anti-bot
499 is unusual: it is not in the HTTP specification, it is not returned to your client, and you usually only see it in someone else's logs. It exists because reverse proxies needed a way to record a specific situation.
Understanding it explains a class of failures that otherwise look like the server being slow.
What 499 means
The widely used definition, popularised by nginx, is: the client closed the connection before the server sent a response.
The crucial detail is the direction. Your client gave up. The server was still working, or was deliberately holding the request, when you disconnected.
That is why you do not receive a 499. By the time the server noticed, you had gone. The status appears in the server's or reverse proxy's access log, not in your response.
Why it matters to a scraper
Two scenarios produce it, and they point in opposite directions.
Your timeout is too aggressive. You set a two-second read timeout, the origin needed four seconds to build a dynamic page, and your client disconnected. The server logs a 499. From your side it looks like a timeout; from theirs it looks like abandonment.
The server is throttling you. Some defences respond by holding the request open, deliberately or as a consequence of load shedding. Your client times out and disconnects, producing the same 499. This is the tarpit pattern described in Anti-Scraping Techniques and How to Respond.
The same log entry, two completely different fixes.
Telling them apart
The distinguishing evidence is the distribution of times.
Uniform slow responses. If almost every request takes roughly the same excessive time, your timeout is probably just too tight for the target's normal latency.
A bimodal distribution. If most requests are fast and a subset hits the timeout exactly, you are being throttled or held. A cluster of durations pinned to your timeout value is a strong signal.
Correlation with volume. If 499s appear only after a certain number of requests in a window, the server is reacting to your rate rather than to the individual request.
Correlation with IP type. If residential addresses are fast and datacenter addresses hang, the network identity is being treated differently, per Datacenter Proxies: Speed vs Detectability.
Setting timeouts properly
The fix for the first case is straightforward, and the structure matters more than the values.
# (connect timeout, read timeout)
timeout = (10, 30)
Separate the two. A connect timeout indicates a network or proxy problem. A read timeout indicates a slow origin. Conflating them hides which one you are hitting, as we note in Using Python requests with Proxies.
Then size the read timeout from measurement rather than from preference. Record the distribution of successful response times, take a high percentile such as the 95th, and set the timeout comfortably above it. A timeout set to the median will fail a large fraction of legitimate requests.
For mobile and residential endpoints, add more headroom. Variable path latency produces false failures on tight timeouts, which we explain in 4G, 5G, and LTE Mobile Proxies Explained.
Responding to the second case
If the evidence points to deliberate holding, increasing your timeout is the wrong move. You would be waiting longer for the same outcome while consuming concurrency.
Better responses:
- Reduce your request rate and see whether the holding stops.
- Reduce concurrency per address, since parallel requests from one IP are a common trigger.
- Rotate between units of work rather than within them, per Rotating vs Sticky Proxies.
- Treat the target as defended and consider a more trusted IP type.
- Abandon the target if holding is consistent and the terms suggest collection is unwelcome. That is a legitimate conclusion, as discussed in Anti-Scraping Techniques and How to Respond.
Instrumenting for a diagnosis
The relevant fields are time to first byte, total duration, status and exit IP. With those, the bimodal-versus-uniform distinction is visible immediately and the diagnosis takes minutes.
Log the timeout value alongside the outcome. It is surprisingly easy to forget that you changed a timeout three weeks ago and to spend an afternoon theorising about defence systems.
Those metrics belong in the dashboard described in Monitoring Scraper Health.
The related statuses
- 504 Gateway Timeout is returned by a proxy when the upstream server did not respond in time. Unlike 499, you receive this one, which means the intermediary gave up rather than you.
- 408 Request Timeout means the server gave up waiting for your request, which is the reverse direction again.
- 503 Service Unavailable is an explicit refusal, often with
Retry-After, covered in Fixing 503 Service Unavailable.
The value of distinguishing them is that each implies a different party broke the connection, and therefore a different fix.
A short summary
A 499 is your client abandoning a request. It is either a timeout you set too tightly or a server holding you deliberately, and the distribution of response times tells you which. Loosen timeouts for the first case; slow down and reconsider the target for the second.