Web Scraping vs APIs
When an official API is available and sufficient, use it. How the two compare on reliability, cost, limits and legal footing, plus the hybrid approach.
- web-scraping
- data-collection
The question comes up on almost every project: the site has an API, so why would you scrape? Sometimes the answer is that you should not. Sometimes the API does not cover what you need.
Getting this wrong in either direction costs money, so it is worth deciding deliberately rather than by habit.
The honest comparison
| Dimension | Official API | Scraping |
|---|---|---|
| Data shape | Documented, structured | Whatever the page contains |
| Stability | Versioned, usually announced | Breaks silently on redesign |
| Authentication | Keys, OAuth, explicit scopes | Whatever the page needs |
| Rate limits | Published, enforced | Informal, discovered by hitting them |
| Cost | Often tiered, sometimes expensive | Proxy and engineering cost |
| Legal footing | Contractual, clear | Depends on terms and jurisdiction |
| Data coverage | Only what the provider exposes | Anything publicly visible |
The two rows that decide most projects are coverage and cost. Everything else is manageable.
When the API is the right answer
The data you need is exposed. Check first. Teams frequently build scrapers for data that a free API tier provides.
The volume fits the tier. If your needs sit inside a reasonable quota, the subscription is usually cheaper than the engineering time and proxy cost of scraping.
You want stability. Documented endpoints with versioning and deprecation notices are far less maintenance than a DOM parser tracking a site's redesigns.
You need a contractual footing. For enterprise use, a service agreement is often a requirement rather than a preference.
When scraping is the right answer
The API does not expose the data. Competitive pricing, assortment coverage, marketplace listings and search results are common examples. This is the core legitimate use case.
The API is priced beyond the value. Vendor pricing for large volumes can exceed the cost of collecting the same public data yourself. That is a commercial judgement, not a technical one.
The data is public and not offered programmatically. Where a site displays information to any visitor but provides no endpoint, collection is the only route.
You need the user-facing view. Prices, ranking and availability as displayed can differ from what an API returns, particularly where personalisation or regional variation is involved. Our notes on that are in Country, State, and City Targeting.
The hybrid that often wins
The framing as a binary is a mistake. A few combinations work well:
API for bulk, scrape for gaps. Use the official endpoint for the data it covers and collect the remainder from the page.
Discover via API, verify via page. Use the API to enumerate items, then check the displayed page for the fields it does not expose.
Scrape the site's own internal endpoint. A public page frequently loads its content from a JSON endpoint the site uses for its own front end. That is smaller and more stable than the rendered HTML, and we cover the technique in Stop Scraping the Page, Find the API Instead.
That last option is worth checking early. It often gives you the stability of structured data with the coverage of the visible page.
Cost comparison, done properly
Comparing a subscription price with a proxy bill is not enough. Include:
- Engineering time to build and maintain the scraper, which dominates over a year.
- Maintenance cost when the site changes, which is unpredictable but certain.
- Proxy cost, based on measured bandwidth rather than a guess. The method is in How Much Residential Bandwidth Do You Need.
- Failure cost, meaning what a missed or wrong data point costs the business. If a pricing decision depends on it, reliability has a monetary value.
A free API with poor coverage may cost more than a paid one. A paid API may cost less than the salary time a bespoke scraper consumes. Work it out with real numbers.
Legal and contractual footing
An API relationship is contractual and usually explicit. Scraping sits on terms of service and, where personal data is involved, data protection law.
The practical points:
- Terms of service are the main exposure, not access statutes. We summarise the case law in Is Web Scraping Legal.
- An API's terms may prohibit using its data to build a competing dataset. Read them rather than assuming.
- Personal data is regulated regardless of source. GDPR applies to scraped personal data exactly as it does to data obtained any other way.
- Rate limits exist for a reason. Respecting them is both polite and self-interested, since ignoring them gets you blocked, as described in Rate Limiting vs Blocking.
A decision checklist
- Does the official API expose the fields I need? If yes, price it and use it.
- If not, is there a public page showing the data? Then find its data endpoint before considering DOM parsing.
- What is the all-in cost of each route, including engineering and failure risk?
- Does the API's terms permit my intended use of the data?
- What is my fallback if the chosen route breaks?
Run that once per data source rather than once per project. Different sources will land on different answers, and a sensible pipeline uses APIs where they fit and collection where they do not.
For the general framing of collection work, see What Is Web Scraping and Data Mining vs Web Scraping.