Scraping Real Estate Listings
Listing structure, search and map endpoints, normalising property records across markets, and the per-market IP requirement for accurate property data.
- real-estate
- web-scraping
Property portals are a well-known collection target and an unusually structured one. Listings share a schema across a portal, which makes parsing straightforward, and the hard parts are geographic coverage, deduplication and data normalisation.
What property data looks like
A listing record has a consistent core, whatever the portal:
- Property identifier and portal source
- Listing type: sale or rent
- Price and currency
- Property type and size in the local unit
- Bedrooms, bathrooms
- Address or location, often approximated on the map
- Agency or agent identifier
- Listing status and first-listed date
- Photo set
Two fields are frequently omitted and matter most for analysis. The first-listed date, without which you cannot tell price reductions from new listings. And the listing status history, which distinguishes a withdrawn listing from a sold one.
Search and map endpoints
Portals expose two collection surfaces, and they return different things.
Search results. Paginated lists filtered by location, price, size and type. The efficient route for coverage, because one request returns many listings. Pagination is bounded, so deep result sets need geographic subdivision rather than page 40.
Map and detail endpoints. Map views often load listings from a JSON endpoint as the viewport moves, which returns a dense set in one request. Detail pages provide the fields search results omit, such as the full description, the photo set and the agent details.
The usual approach is search for discovery, then detail requests for the listings that need full records, which keeps the request count proportional to the listings you actually keep.
Market-by-market collection
Property data is inherently local. Prices are in the local currency, sizes are in the local unit and the listing conventions differ.
| Property | Why it differs | What to store |
|---|---|---|
| Currency | Local pricing | Currency code on every record |
| Area unit | Square metres versus square feet | Unit plus the original numeric value |
| Price convention | Some markets quote monthly, or per square metre | Price period as a field |
| Location | Postcode, district or region systems | Original text plus a normalised region |
| Portal | Coverage differs by country | Portal and market as dimensions |
Collecting one market and comparing the values to another without normalising is the most common analytical error. Store the original alongside the normalised value, so an error in normalisation is recoverable.
The per-market IP requirement
Portal results are geo-localised. Some portals show different listings, prices or availability depending on where the request originates, and some restrict non-local access entirely for regulatory reasons.
The implication: an address in the target market, held stable for the duration of a market's collection run. Hosting ranges are both more likely to be challenged and more likely to receive different content, which is the general case in Why Residential IPs Are Trusted.
For a multi-market operation, one address per market on a stable session is the cleanest model, and the same reasoning that applies to localised pricing collection applies here.
Deduplication
The same property appears on several portals and often several times on one portal, because agents relist rather than update.
Three signals, in order of reliability:
Identifier. The portal's own listing id, which changes on a relist.
Address plus attributes. Postcode, street normalised, plus bedrooms and approximate size. Catches relists across portals.
Photo hashes. A perceptual hash of the primary photo catches the same listing reused with a different description.
Combine address matching with photo hashing and you catch most duplicates. Without deduplication, a market's active-listing count is inflated by an amount that varies by city, which quietly invalidates any supply or demand conclusion drawn from it.
Normalising records
The output schema should be portal-independent, so records from several sources are comparable:
Market, Portal, ListingId, ListingType, Price, Currency, PricePeriod,
AreaValue, AreaUnit, Bedrooms, Bathrooms, Region, PostalCode, Lat, Lon,
FirstListed, Status, AgentId, SourceUrl, CollectedAt
Two rules keep it usable. Never overwrite the original values with normalised ones, and always store CollectedAt, because property data is a time series and a record without a timestamp cannot be placed in it.
Cadence
Property portals tolerate slow collection and penalise fast collection. The data also changes slowly: new listings appear daily, prices change less often, and status changes lag.
A daily run for new-listing detection and a weekly run for full re-collection covers most use cases. Running hourly gains nothing the site's own freshness does not already limit, and it dramatically increases the chance of a block.
Terms and personal data
Portal terms usually restrict automated collection, and agent contact details are personal data in most jurisdictions. Collect the property facts, avoid scraping agent contact details you have no basis for, and check the terms for the market before starting. The framework is in Data Collection Ethics for Engineering Teams and the legal background in Is Web Scraping Legal?.
For the address layer, see Residential Proxies for Real Estate and Residential.