Monitoring Scraper Health

Pipelines rarely fail suddenly. They degrade: a selectors drifts, an IP range gets flagged, a target tightens its defences, and the output becomes progressively worse while the job continues to exit successfully.

Monitoring is what converts that slow decay into an alert you can act on.

The metrics that matter

Six, in rough order of usefulness.

Success rate. Validated results divided by attempts. Validate against a schema rather than a status code, because a challenge page is a successful HTTP response with no data in it. The definition matters, and it is the argument in Latency vs Success Rate.

Record count per run. The simplest and often earliest signal. A drop from 4,200 to 3,100 records is visible immediately and needs no schema work to measure.

Block and challenge rate. 403s, 429s and interstitials per hundred requests. This is your proxy-quality metric, covered in Captchas and Proxies.

Latency percentiles. Track p50 and p95 rather than the mean. Averages hide the tail that causes timeouts, and a rising p95 is an early sign of throttling.

Coverage. Collected items against what the target reports. If the site says 4,200 listings and you have 3,100, you are missing a quarter of the data and may have no idea which quarter. The detection method is in Handling Pagination in Scrapers.

Freshness. The age of the newest record. A pipeline that runs on schedule but returns stale data has quietly stopped working.

Metrics that look useful but are not

Requests per second. Interesting for infrastructure, uninformative about data quality. A high request rate with a low success rate is worse than the reverse.

Uptime. A scraper has no meaningful uptime. It either produces data or it does not.

Average latency. The mean hides the distribution, and the distribution is what times out.

Bytes transferred. A cost input rather than a health metric, useful for budgeting per What Is Proxy Bandwidth.

Instrumenting the fetch layer

Everything above comes from logging enough per request. Six fields cover most needs.

log.info(
    "fetch url=%s status=%s ok=%s exit_ip=%s session=%s duration_ms=%d bytes=%d",
    url, response.status_code, is_valid, exit_ip, session_id,
    int(duration * 1000), len(response.content),
)

Two of those are easy to omit and hard to work without later.

The exit IP lets you attribute a failure to an address rather than guessing. A cluster of failures on one IP is a proxy problem; failures spread across all IPs are a target or client problem.

The validity flag, separate from the status code, is what makes success rate meaningful.

Alerting that earns its place

Alert on changes, not on absolutes, and keep the number of alerts small enough that each one is taken seriously.

Page the person on call for: zero records returned, or a job that has not completed within its window.

Send a daily digest for: a success rate below a threshold, a coverage gap, a rising block rate, or a freshness breach.

Do nothing automated, just record for: latency percentiles and per-endpoint success rates. These inform investigation rather than demanding action.

Thresholds should come from the pipeline's own history rather than from a guess. If a job normally runs at 96 percent success, alert below 90 percent. If it normally takes 20 minutes, alert above 40.

Per-endpoint health

When you run a pool, aggregate metrics hide a failing address. Track success rate and latency per endpoint, and retire addresses that degrade, as described in What Is a Proxy Pool.

This also distinguishes the two common diagnoses:

• Several endpoints degrading together means the target tightened its defences or your behaviour changed.
• One endpoint degrading means that address is burned, and the remedy is a cooldown rather than a pipeline change.

Distinguishing failure modes

A dashboard is only useful if the signals separate the causes. Three checks resolve most incidents quickly:

Compare a direct request. If a non-proxied request also fails, the target is down or the resource is gone.

Test a different IP type. If a residential endpoint succeeds where datacenter fails, the network identity is the cause, per Datacenter Proxies: Speed vs Detectability.

Compare the raw response. If the page is fetched but your fields are empty, the parser drifted rather than the network failing. That is the failure the fixtures in Parsing HTML and JSON Reliably are designed to catch.

Where to put the data

Two options that work.

A time-series store plus a dashboard. Record each run's metrics and chart them over time. Prometheus and Grafana, or a hosted equivalent, are the standard stack.

A metrics table in your existing database. Perfectly adequate for smaller pipelines, and it keeps the monitoring next to the data. One row per run with the fields above, queried on demand.

The second is often the right starting point, because it takes an afternoon and covers the cases that actually occur. Add the dashboard when the table stops being enough.

The habit that matters

Log per request, aggregate per run, and alert on change. Every failure mode above becomes diagnosable in minutes rather than by trial and error, and the metrics double as the evidence base for the proxy and client decisions made elsewhere on this blog.

For the collection side the metrics describe, see What Is Web Scraping, and for the proxy decisions they inform, Choosing a Proxy for Web Scraping.