News Scraping with Python

News is an appealing collection target: it is public, it is text-heavy rather than media-heavy, and it changes on a predictable schedule. It is also messier than it looks, because the same story appears in many places with different framing, different timestamps and different URLs.

This is a practical approach that starts with the easiest source and escalates only when necessary.

Start with RSS and sitemaps

Most outlets publish an RSS or Atom feed, and many publish a news sitemap. Both are designed for machine consumption, which makes them faster, lighter and more stable than scraping rendered pages.

import feedparser

feed = feedparser.parse("https://example.com/feed")
for entry in feed.entries:
    record = {
        "title": entry.title,
        "url": entry.link,
        "published": entry.get("published"),
        "summary": entry.get("summary"),
    }

Feeds usually carry a truncated summary rather than the full article, so you use them for discovery and fetch the article only when you need the body. That split alone saves most of the bandwidth, which matters because the cost model is per gigabyte, as explained in What Is Proxy Bandwidth.

Also check the site's robots.txt for a sitemap reference, which often lists recent articles more completely than a feed.

Fetching article pages

For the body text, a plain HTTP client is usually sufficient because news articles are typically server-rendered.

import requests

response = requests.get(url, timeout=(10, 30), headers={"User-Agent": "..."})
response.raise_for_status()
html = response.text

If the outlet renders client-side, which is common among newer publishers, you need a browser. The trade-off is in Playwright vs Selenium, and the lighter path in Web Scraping with Python.

Extracting the article body

Extracting the main content, rather than the whole page, is a solved problem worth using rather than reimplementing. Readability-style libraries remove navigation, sidebars, ad slots and comments.

from readability import Document

doc = Document(html)
title = doc.short_title()
body_html = doc.summary()  # main content, boilerplate removed

If you prefer to control the extraction, target the article container and select on semantic elements such as article, time and the Open Graph metadata, which outlasts class names. The selector discipline is in Parsing HTML and JSON Reliably.

Extract these fields, which are the ones analysis actually needs:

• Title, from the page or from Open Graph.
• Canonical URL, because many outlets publish under multiple paths.
• Publication timestamp and timezone, both raw and normalised.
• Author byline, where present.
• Body text, stripped of boilerplate.
• Section or category, often in the URL or breadcrumb.
• Outlet, so aggregation knows the source.

Normalising publication time

Timestamp handling is where news pipelines most often go wrong. Three problems recur:

Timezones. A feed may publish in UTC while the page displays local time. Parse to a timezone-aware value and store the original string.

Updated versus published. Some outlets update articles substantially. Keep both fields, because an updated timestamp does not mean the story is new.

Date-only feeds. Some feeds omit the time, which makes intra-day ordering impossible. Record that limitation rather than inventing a time.

Normalising these is the same discipline as currency handling in Residential Proxies for Travel Fare Aggregation: convert, keep the original, and document the assumption.

Deduplicating coverage

The same story reaches the wire and appears across dozens of outlets within minutes. For most analysis, you want to group those rather than count them as separate events.

Practical approaches:

• URL canonicalisation first, which removes tracking parameters and resolves syndication paths.
• Title similarity, using a token-based measure, to group near-identical headlines.
• Content shingling or a hash of a normalised paragraph, which catches rewrites that keep the core text.
• Time proximity, since the same story clusters within a short window.

Store the cluster identifier alongside the article, so an outlet can be counted once per story rather than once per publication. This is the deduplication principle from Handling Pagination in Scrapers, applied at the corpus level.

Politeness and paywalls

Two ethical and practical boundaries matter here.

Paywalls. Many outlets meter access. Circumventing a paywall is not a legitimate use of scraping infrastructure, and doing it with proxies turns a technical question into a legal one. Collect what is publicly served to an anonymous visitor, and if the article is gated, do not attempt to work around it.

Volume. News sites are load-sensitive during breaking events, which is exactly when you might be crawling most aggressively. Pace requests, honour rate limits, and treat a 429 as a signal to slow down, per Rate Limiting vs Blocking.

The wider framework is in Data Collection Ethics for Engineering Teams, and the legal context in Is Web Scraping Legal.

Monitoring the pipeline

News collection fails in characteristic ways, and each has a cheap early signal:

• Feed stops updating, which usually means a URL change rather than no news.
• Article count per run drops, which points at a parser change.
• Body extraction returns navigation text, which means the main-content selector has drifted.
• Duplicate rate spikes, which suggests your canonicalisation broke.

Track all four and alert on movement. The general approach is in Monitoring Scraper Health.

Where proxies fit

For a handful of outlets fetched once an hour, you may not need a proxy at all. At scale, across many outlets and more frequent polling, proxies help with rate limits and allow you to check what a reader in a specific market sees. Confirm any endpoint with the proxy checker before scaling, and the proxy selection logic is in Choosing a Proxy for Web Scraping.