E-Commerce Data Scraping: Methods and Tools

Retail data collection is the most common commercial scraping task and one of the most heavily defended. Success is less about the parser and more about the request layer.

What is worth collecting

Five data classes, each with a different cadence and a different collection cost.

Price. The core signal. Needs the product identifier, the price, the currency, the timestamp, and the region it applies to. Without region and timestamp the number is not comparable to anything.

Stock and availability. Present or absent, and any low-stock indicator. Price without availability is misleading, since an unavailable product's price is not a market price.

Assortment. Which variants exist: size, colour, capacity. Changes in assortment are a leading indicator of a product transition.

Ratings and review counts. Aggregate values are cheap; review text is not, and the volume is often capped.

Promotions. Banner and badge states, discount percentages, bundle offers. Often rendered client-side and easy to miss.

The defences you will meet

Retail sites are the most aggressive category, because price data has obvious competitive value.

The two that surprise people most are client-side rendering, which makes the HTML useless, and geo-localisation, which makes the data wrong rather than missing. The first is covered in Scraping Browsers Explained, and the second is a proxy requirement.

The collection layers

A retail collector has four independent parts, and only one of them is parsing.

1. Request layer. The HTTP client, the address, the session handling and the retry policy. This is where most failures happen.

2. Rendering decision. Whether the data is in the initial HTML or needs a browser. Check the site's network requests before assuming you need a full browser, because the JSON API is often available and far faster.

3. Parser. Extraction and validation against a schema. Validate on the way in rather than discovering three weeks later that a field has been null since a redesign.

4. Storage and change detection. Storing history rather than current state, so that changes and their timing are visible.

A minimal example

The request layer is ordinary Python, and the proxy configuration is the only unusual part.

import httpx

def fetch(product_url: str, market_proxy: str) -> dict:
    with httpx.Client(proxy=market_proxy, timeout=20.0) as client:
        response = client.get(
            product_url,
            headers={"Accept-Language": "pl-PL,pl;q=0.9"},
        )
        response.raise_for_status()
        return parse_product(response.text)

Four properties of that snippet matter more than the code itself. The proxy is market-specific rather than generic. The timeout is set, because a hanging request blocks a pool. The status is raised rather than ignored, so a 403 is not parsed as a page. And the language header matches the address, so locale and network do not contradict each other.

The proxy layer

This is what determines whether the collector scales.

Datacenter addresses for public catalogue reads at volume, where the data is not personalised and the site tolerates it.

Residential addresses where pricing is localised, where the site blocks hosting ranges, or where the data must reflect real regional pricing.

Mobile addresses only where a session needs high trust, which is rare for public retail data.

The allocation is the same pattern described in A Proxy Strategy for E-Commerce Teams: cheap capacity where the data is public and static, higher-trust capacity where locality or blocking requires it.

Verification and cadence

Two operational notes that decide whether the data is trustworthy.

Verify the address per market. A geo-targeted fetch is only correct if the exit address is in the target country. The Proxy Checker and IP Lookup confirm this, and a mismatch silently produces another market's prices.

Match cadence to the decision. A price checked hourly for a product with monthly price changes wastes budget; a price checked weekly for a product that changes daily is useless. Cadence should follow the volatility of the category.

Rules and limits

Public retail data is generally collectable, but the terms of service and the rate at which you collect are the real constraints. Keep request rates within what a site would tolerate from a human user, respect robots directives where they apply to your use, and do not collect personal data you have no basis for. The framework is in Data Collection Ethics for Engineering Teams and the legal background in Is Web Scraping Legal?.

For the product layer, Residential covers the localised case, and for the solution context, E-Commerce.