Parsing HTML and JSON Reliably

Fetching data is the easy half. Turning a page into records you can trust is where quality is won or lost, and the failure mode is rarely an exception. It is a dataset that looks fine and is quietly wrong.

This is how to build parsing that fails loudly instead of silently.

Select on meaning, not on position

The most common cause of a parser breaking is a selector that describes today's layout. div > div:nth-of-type(3) > span is a statement about markup, not about the data you want.

Prefer, roughly in order of durability:

1. Structured data, such as JSON-LD or microdata embedded for search engines.
2. Stable semantic attributes, such as data-testid, data-price, itemprop or a meaningful id.
3. Semantic elements, such as table, time or an element with descriptive attributes.
4. Class names, which are fragile because build tools generate them.
5. Position, which is a last resort.

price = soup.select_one("[itemprop='price']") or soup.select_one("[data-price]")

Trying a couple of selectors in order costs nothing and buys resilience against partial redesigns.

Anchoring to a container

Select the smallest enclosing element and extract fields within it, rather than finding "the third price on the page". Container-scoped extraction survives reordering, which global selection does not.

rows = []
for card in soup.select("[data-testid='product-card']"):
    rows.append({
        "name": text_of(card.select_one("[data-testid='product-title']")),
        "price": text_of(card.select_one("[data-price]")),
    })

Validate before you trust

This is the step teams skip and then regret. A page that loads but has moved its price element produces a record with a missing field. Without validation, that flows into analysis as a zero or a null and skews the result.

from pydantic import BaseModel, field_validator

class Product(BaseModel):
    name: str
    price: float
    currency: str = "USD"

    @field_validator("price")
    @classmethod
    def price_must_be_positive(cls, v: float) -> float:
        if v <= 0:
            raise ValueError("price must be positive")
        return v

accepted, rejected = [], []
for row in rows:
    try:
        accepted.append(Product(**row))
    except ValidationError as exc:
        rejected.append({"row": row, "error": str(exc)})

Two habits in that snippet matter more than the library choice.

Keep the rejects. Do not discard invalid records silently. A rising reject count is your earliest warning that the page changed or that you are being served a blocked variant.

Validate plausible values, not just types. A negative price, a date in the future or a zero quantity are all signals that the extraction grabbed the wrong thing. Type checking alone would have accepted them.

The equivalent in other stacks is Zod in TypeScript, JSON Schema generally, or dataclasses with explicit checks. The principle is identical.

Normalising values

Raw text is rarely analysis-ready. Normalise at the boundary, and keep the raw value when the normalisation is lossy.

import re
from decimal import Decimal

def to_decimal(raw: str | None) -> Decimal | None:
    if not raw:
        return None
    cleaned = re.sub(r"[^0-9.,]", "", raw)
    # Locale-aware: decide the decimal separator before stripping separators.
    cleaned = cleaned.replace("\u00a0", "")
    if "," in cleaned and "." in cleaned:
        cleaned = cleaned.replace(",", "")
    elif "," in cleaned and len(cleaned.split(",")[-1]) == 2:
        cleaned = cleaned.replace(",", ".")
    try:
        return Decimal(cleaned)
    except Exception:
        return None

Currency and locale handling is where naive parsing produces quietly wrong numbers, which is why we treat it as a first-class concern in Residential Proxies for Travel Fare Aggregation and Residential Proxies for Price Monitoring.

Dates deserve the same treatment: parse to a canonical timezone-aware format, and store the original string. Timezone confusion is a common source of off-by-one-day errors.

JSON: validate the shape, not just the parse

response.json() succeeding means the body was valid JSON, not that it has the fields you expect. Validate the structure.

payload = response.json()
items = payload.get("data", {}).get("items")
if not isinstance(items, list):
    raise ValueError("unexpected payload shape")

Defensive access with .get() and explicit type checks turns a confusing KeyError deep in the pipeline into a clear failure at the boundary.

Keeping the parser honest

A parser needs tests, and the tests need fixtures. Save a real page and assert your extraction against it.

• One fixture per page type, so a redesign in one template is caught specifically.
• Assert on values, not just presence, so a selector grabbing the wrong element fails.
• Include edge cases: a product with no price, a listing with one result, a page with unusual characters.

When a target changes, a failing fixture tells you exactly what broke in seconds rather than after a dashboard fills with nulls. That is the same discipline as the alerting described in Monitoring Scraper Health.

The rule to remember

Parse leniently, validate strictly, keep the rejects. Fetching and extracting are best-effort; the boundary where data enters your database is not.