Web Scraping with Python: Complete Guide

Python is the default language for scraping because the ecosystem is mature and the code is short. This guide builds a real scraper in stages, from a single page to something you could schedule, adding proxies at the point where they become necessary.

Stage 1: fetch a static page

For server-rendered HTML, requests is enough.

import requests

response = requests.get(
    "https://example.com/products",
    headers={"User-Agent": "Mozilla/5.0 (compatible; research-bot)"},
    timeout=20,
)
response.raise_for_status()
html = response.text

Three details in those six lines are worth noting. Setting a user agent identifies your client honestly rather than pretending to be a browser you are not. A timeout prevents a hanging request from stalling the job. And raise_for_status turns an error response into an exception rather than letting you parse an error page as if it were data.

Stage 2: parse the HTML

BeautifulSoup handles HTML, including the malformed variety found in the wild. Python's standard library also includes html.parser; lxml is faster if you have it.

from bs4 import BeautifulSoup

soup = BeautifulSoup(html, "html.parser")

products = []
for card in soup.select("[data-testid='product-card']"):
    name = card.select_one(".product-name")
    price = card.select_one("[data-price]")
    products.append(
        {
            "name": name.get_text(strip=True) if name else None,
            "price": price["data-price"] if price else None,
        }
    )

Two habits make this survive layout changes. Selecting on stable attributes such as data-testid or structured markup survives redesigns far better than relying on element order or generated class names. And defaulting missing fields to None means one absent element does not crash the run, which is the discipline we describe in Parsing HTML and JSON Reliably.

Stage 3: validate before you trust

Do not let unvalidated output flow downstream. A schema check catches the case where the page loaded but the price element moved, which otherwise looks like a product with no price.

from pydantic import BaseModel, ValidationError

class Product(BaseModel):
    name: str
    price: float

valid = []
for row in products:
    try:
        valid.append(Product(**row))
    except ValidationError:
        continue  # record the failure, do not silently drop the page

Recording failures rather than discarding them matters later: a rising failure rate is your earliest signal that a page changed or that you are being blocked.

Stage 4: add politeness

A scraper that hammers a server is both rude and self-defeating. Add a delay between requests, and back off on rate-limit responses.

import time
import random

time.sleep(random.uniform(1.0, 2.5))  # jittered, not fixed

Jitter matters. A machine-regular interval is itself a detection signal, which we explain in How Anti-Bot Systems Detect Scrapers. Honour Retry-After when a server provides it, as described in Rate Limiting vs Blocking.

Stage 5: when you need a browser

If the content is built by JavaScript, an HTTP client returns an empty shell. Two options:

• Find the underlying API, which is faster and more stable when it exists. The technique is in Stop Scraping the Page, Find the API Instead.
• Use a browser engine. Playwright and Selenium both work in Python; the trade-offs are in Playwright vs Selenium and the proxy setup in Playwright Scraping with Proxies.

Browser automation is slower and heavier, so reach for it when you have established that you need it.

Stage 6: add a proxy

This is when the target starts rate limiting or classifying you. The configuration is a dictionary.

import os

proxy = os.environ["LB_PROXY_URL"]  # http://user:pass@host:port

session = requests.Session()
session.proxies = {"http": proxy, "https": proxy}
session.headers["User-Agent"] = "Mozilla/5.0 (compatible; research-bot)"

response = session.get("https://example.com/products", timeout=20)

Reusing a Session keeps the connection alive across requests, which matters when you send many. Per-request rotation, pooling and retry policy are covered in What Is a Proxy Pool.

Confirm the proxy is working before you build on it: the proxy checker reports the exit IP and added latency, and IP lookup confirms the location if you are using targeting.

Stage 7: run it on a schedule

Separate the fetch logic from the scheduling. Where to run it, how to store secrets and how to alert on failures are covered in Scheduling Scrapers with CI and Monitoring Scraper Health.

What to build next

Once a scraper works on one page, the work is reliability: handling pagination, deduplicating, retrying intelligently and keeping the parser in sync with the site. The pieces are in Handling Pagination in Scrapers and Parsing HTML and JSON Reliably.

If you are new to the whole area, start with What Is Web Scraping for the context, and keep the practical limits in mind by reading Is Web Scraping Legal.