Scraping Classified Listings

Classifieds are the easiest data to parse and the hardest data to keep clean. The markup is simple, the volume is enormous, and the same item reappears every few days because posting is free.

The structure

A classified site organises by region, then by category and subcategory, with paginated result lists and individual listing pages.

Result list. Title, price when present, location, and a relative posting time such as "3 days ago". The relative timestamp is the trap: it must be converted to an absolute date at collection time, or the record ages inconsistently.

Listing page. Description, images, the posting date, an identifier in the URL, and often a contact method rendered as an image or behind a form.

Three fields decide whether a collection is useful:

Absolute posting date, resolved at collection time.
Listing identifier, from the URL, which is stable per post.
Region and category, which are the natural partition keys.

Scoping by city

Classifieds are partitioned by region, and the region is the natural unit of collection. Attempting a national sweep produces an enormous result set, most of which you do not need, and a request pattern no user would generate.

Collect region by region, with an address in or near the region. Two reasons:

Content accuracy. Some sites vary what they show and how they rank by request location, so results from a distant address may not match the local view.

Block behaviour. A large national sweep from one source is the pattern that gets rate-limited first, per Rate Limiting vs Blocking.

Region-scoped collection with one address per region also makes the data easier to reason about, because every record already carries its market.

Deduplication is the main problem

Posting is free, listings expire on a timeout, and posters repost rather than renew. The result is that a single item can appear dozens of times over a year.

Matching strategy, in order of reliability:

Listing identifier. Exact, but new on every repost.

Text similarity. Normalise the title and description, then compare using a similarity measure rather than equality, because posters make small edits. A token-based similarity like Jaccard or a shingle comparison handles this reasonably.

Image hash. A perceptual hash on the primary image catches a repost with a rewritten description.

Attribute match. Price, category, region and size in combination.

In practice, title similarity plus image hash catches most duplicates. Decide what counts as the same listing and store the grouping rather than deleting, so the repost history remains available, because for many analyses the repost frequency is the signal.

The time-series angle

For classifieds, the interesting data is usually change over time rather than a snapshot. Three derived measures worth computing at collection time:

First seen and last seen per logical listing.
Repost count over a window, which is a rough measure of how hard something is to sell.
Price trajectory across reposts, since a falling price across successive posts is a strong signal.

None of these are available from a single collection. They require the same listing to be tracked across runs, which means the deduplication grouping key must be stable.

Request rates and the address layer

Classified sites are lightly defended compared to retail. There is usually no bot management, and the limit that matters is request rate.

Practical settings:

Low concurrency. A handful of parallel requests at most.
Pauses between pages. A second or more, varied rather than fixed.
Residential addresses if collection runs at any volume, because a single address making thousands of daily requests is the pattern that draws attention. The reasoning is in Why Residential IPs Are Trusted.
Rotation is acceptable for list pages, since there is no session, but hold an address for a region rather than rotating between requests, so the vantage point is consistent.

The general principle is that the site tolerates what a human would do. A user browsing classifieds loads a page, reads, and clicks. A collector should not exceed that shape by an order of magnitude.

Privacy and personal data

Classified listings frequently contain personal information: names, phone numbers, addresses, sometimes photos of identifiable people or interiors. This is the category where collection is most likely to cross into personal data, and the obligations attach to the data regardless of how publicly it was posted.

Practical rules:

Collect the fields your analysis needs, and not the contact details by default.
Do not build profiles of individual posters.
Check the site's terms and the local rules for personal data, particularly under GDPR if any subjects are in the EU.
Set a retention limit and delete raw records you no longer need.

The framework is in Data Collection Ethics for Engineering Teams and the legal background in Is Web Scraping Legal?.

For the technical side of scaling, see E-Commerce Data Scraping and What Is a Proxy Pool.