What Is Web Scraping? A Practical Introduction
A grounded introduction to web scraping: what it is, how it differs from an API, the basic anatomy of a scraper, and where the legal and ethical lines sit.
- web-scraping
- data-collection
Web scraping is the practice of having software fetch a web page and extract structured information from it, usually at a scale that manual reading cannot match. Instead of copying prices into a spreadsheet by hand, you write a program that does it thousands of times.
That is the whole idea. Everything else is engineering and judgement.
What scraping is used for
Common and generally defensible uses include price monitoring, assortment tracking, real-estate listing collection, travel fare comparison, market research, ad verification, brand protection and academic research on public data.
The common thread is that the data is displayed publicly, and the collector wants it in a form they can analyse rather than read.
How it differs from an API
An API is a documented interface the site offers for programmatic access. It returns structured data, has stated limits, and its terms are usually explicit. Where an API exists and covers your need, use it.
Scraping is for what the API does not expose: a competitor's pricing page, a marketplace listing, a search result page. It returns whatever the web page contains, so you have to extract the fields yourself and maintain that extraction as the page changes.
The trade-offs are compared properly in Web Scraping vs APIs, including the hybrid pattern where a site's own front-end calls an internal JSON endpoint that is cheaper and more stable than the rendered page.
The anatomy of a scraper
Every scraper has three parts.
Fetch. Retrieve the page. For simple static HTML this is an HTTP request. For pages that build their content with JavaScript, you need a browser engine, which we compare in Picking a Browser Automation Tool.
Parse. Extract the fields you want. Libraries such as BeautifulSoup in Python handle HTML parsing, and the durability techniques are in Parsing HTML and JSON Reliably. Writing selectors that survive a layout change is the part that takes practice.
Store. Write the result somewhere useful, with enough metadata to make it auditable: the URL, the timestamp and the observed values.
That is the minimum. Real pipelines add scheduling, retries, deduplication, validation and monitoring, which is why we treat scraping as a data pipeline rather than a script. See Monitoring Scraper Health.
Where proxies enter
Scraping works fine without a proxy until a target objects. The objections are predictable:
- Rate limiting, where a site slows or refuses requests from one address after a threshold.
- Bot classification, where a site decides your requests are automated and blocks them.
- Geo-dependency, where the content you want differs by location.
A proxy addresses all three, and which type you need depends on the cause. Choosing a Proxy for Web Scraping works through the decision. For a first project, though, note that a polite scraper on a moderate target may need no proxy at all.
Being a good citizen
Technical capability is not permission. A few habits make scraping sustainable:
- Respect rate limits and honour
Retry-Afterwhen a server sends it. The signals are in Rate Limiting vs Blocking. - Read the terms and the
robots.txtfile, and treat both as constraints rather than obstacles. - Keep concurrency low and pace requests, so your load is indistinguishable from normal traffic.
- Collect the minimum you need. If a price is all you want, do not keep everything else.
- Do not collect personal data without a lawful basis. This is the sharpest legal line and it is worth taking seriously.
The legal picture in one paragraph
In the United States, the Ninth Circuit held in hiQ Labs v. LinkedIn (938 F.3d 985, affirmed on remand in April 2022) that scraping publicly accessible data does not violate the Computer Fraud and Abuse Act, while the Supreme Court in Van Buren v. United States, 593 U.S. 374 (2021), narrowed "exceeds authorized access" to a gates-up-or-down test. Notably, hiQ still lost on breach-of-contract and trespass claims, which is the practical warning: contract and terms of service are the real exposure, not just the access statute. In the EU, GDPR governs personal data regardless of how it is collected.
That is a summary, not legal advice. Our fuller treatment is in Is Web Scraping Legal, with a practical framework in Data Collection Ethics for Engineering Teams.
A first project
Start small and specific. Pick one public page, extract three fields, store them with a timestamp, and run it daily for a week. Then add the pieces you actually needed: probably retries, probably a schema check, and possibly a proxy once the target starts objecting. The Python walkthrough is in Web Scraping with Python, and the tooling is on the proxy checker side when you get there.