Collecting Amazon Product and Review Data

Amazon is the most requested retail target and the one where naive collections fail fastest. Two properties explain almost every problem: the data is per-marketplace, and the rate limits are enforced aggressively on both requests and sessions.

Marketplace first, product second

The same ASIN returns different prices, currencies, availability and sometimes different titles depending on the marketplace domain. There is no single Amazon price.

That means the marketplace is a dimension of every record, not a detail:

Collecting "the Amazon price" for a product means collecting it per marketplace, from an address in that marketplace's country, and storing the domain alongside the value. A US price obtained through a Polish address is either wrong or a redirect, which is the general localisation issue in E-Commerce Data Scraping.

What the pages give you

Product page. Title, ASIN, current price, list price when discounted, availability, seller, rating average, review count, variant options and the buy-box state. The buy box matters for anything commercial, because the price shown is the buy-box seller's.

Search results. ASIN, title, price, rating and sponsored status. Useful for assortment monitoring, and faster to collect in bulk than individual product pages.

Review pages. Individual reviews with rating, date, verified-purchase flag, variant and text.

Review pagination is capped

Amazon does not expose the full review history through pagination. In practice, review pages are limited to a small number of pages per product (commonly around ten pages of ten reviews, so on the order of a hundred reviews), regardless of the review count on the product page.

Three consequences:

• You cannot collect complete review history by paging. The review count and the accessible reviews are different numbers.
• Collect incrementally. Run frequently and store diffs, so you accumulate coverage over time rather than trying to fetch everything at once.
• Do not treat the sample as representative. The most recent reviews are over-represented, which biases sentiment analysis toward recent problems or recent campaigns.

For products where complete review history matters, the official Product Advertising API is the appropriate route where it covers the need.

Rate limits and blocking

Amazon's defences escalate quickly and in a recognisable order.

1. Throttling. Latency rises, then 503 responses appear.
2. CAPTCHA interstitials. Requests are served a challenge page instead of the product. Handled per How to Bypass CAPTCHA, which is mostly about avoiding the need.
3. 403 responses from hosting ranges, described in Fixing 403 Forbidden.
4. Session invalidation, where cookies stop working mid-collection.

The response is the standard one: fewer requests per address, more addresses, a plausible locale per address, and a slower cadence. Amazon notices volume per source before it notices anything else.

The proxy layer

Three requirements, all of which are about matching the marketplace rather than defeating a defence.

Country match. The address must be in the marketplace's country, or the collection returns another marketplace's data.

Residential or ISP addresses. Hosting ranges are blocked more often and are served different content when they are not.

Session discipline. Product pages tolerate rotating addresses; anything involving a session, a cart or a logged-in view needs a sticky session.

The allocation logic is in A Proxy Strategy for E-Commerce Teams and the address comparison in Datacenter vs Residential vs Mobile Proxies.

Start with the official API

Before building a collector, check whether the Product Advertising API covers what you need. For a catalogue of your own or associated products, prices, availability and some review metadata, it is the correct route and it removes the blocking problem entirely.

It does not cover everything. Competitor pricing at scale, sponsored placement, buy-box history and full review text typically require collection from the site, and each of those carries the rate-limit constraints above.

Practical pattern

For a price and review monitor across several marketplaces:

1. Store the marketplace as a first-class field on every record.
2. Use one address per marketplace, in that country.
3. Collect product pages on a schedule matched to price volatility, not to convenience.
4. Accumulate reviews incrementally rather than attempting complete history.
5. Validate every parsed record against a schema, so a layout change fails loudly instead of writing nulls.
6. Track success rate and block rate per marketplace, as described in Monitoring Scraper Health.

Terms and limits

Amazon's conditions of use restrict automated collection, and the rate at which you collect is a real constraint independent of the technical one. Keep request rates within what the site would tolerate from a human user, do not create accounts for collection, and do not collect personal data from reviews. The framework is in Data Collection Ethics for Engineering Teams.

For the monitoring context, see Price Monitoring Tools Compared and E-Commerce.