Puppeteer Web Scraping: A Practical Guide

Puppeteer is the Node ecosystem's browser automation library, and it remains a solid choice for scraping in JavaScript and TypeScript projects. It is lighter than writing a full Playwright setup if your stack is already Node, and its API is compact.

The trade-offs are worth knowing before you commit.

Launching with a proxy

Proxy settings go in the launch arguments, the same mechanism Chrome itself uses.

import puppeteer from "puppeteer";

const browser = await puppeteer.launch({
  headless: true,
  args: [
    "--proxy-server=http://proxy.lightningbytes.com:1080",
    "--no-sandbox",
  ],
});

const page = await browser.newPage();
await page.goto("https://api.ipify.org");
console.log(await page.evaluate(() => document.body.innerText));
await browser.close();

--no-sandbox is commonly needed in containers. Understand what it disables before using it in a context where you do not control the input.

The authentication gap

Like Selenium, Puppeteer has no clean path to authenticated proxies, because the browser does not expose an automatable login prompt. Your options are the same three:

IP allowlisting, which is the simplest when your workers have stable outbound addresses.

page.authenticate(), which handles HTTP authentication for the page context and works for some proxy setups:

await page.authenticate({ username: "lb-USERNAME", password: "SECRET" });

Coverage of this varies, so test it before relying on it.

A local relay that adds credentials and forwards to the gateway, which is the most portable option.

We compare the approaches in Proxy Authentication and Selenium Proxy Setup and Rotation.

Waiting for the right signal

Puppeteer's older waiting shortcuts were deprecated, and the modern approach is explicit.

await page.goto("https://example.com/products", { waitUntil: "domcontentloaded" });
await page.waitForSelector("[data-testid='product-card']", { timeout: 20000 });

Prefer domcontentloaded over networkidle for scraping. networkidle waits for the network to go quiet, which on modern pages with polling or analytics may never happen, producing timeouts on pages that had the data you wanted several seconds earlier.

Extracting data

page.evaluate runs code in the page context, which is the most efficient extraction route because it avoids round trips per element.

const products = await page.evaluate(() => {
  return [...document.querySelectorAll("[data-testid='product-card']")].map((card) => ({
    name: card.querySelector("[data-testid='product-title']")?.textContent?.trim() ?? null,
    price: card.querySelector("[data-price]")?.getAttribute("data-price") ?? null,
  }));
});

Two habits that matter: select on stable attributes such as data-testid rather than generated class names, and tolerate missing elements with optional chaining so one absent node does not fail the whole extraction. The reasoning is in Beautiful Soup: Parsing HTML Without the Headache, and the validation step in Parsing HTML and JSON Reliably.

Cutting bandwidth and load time

Rendered pages pull far more than the data you want. Request interception lets you block the rest.

await page.setRequestInterception(true);
page.on("request", (request) => {
  if (["image", "media", "font", "stylesheet"].includes(request.resourceType())) {
    request.abort();
  } else {
    request.continue();
  }
});

Verify that your selectors still work after blocking stylesheets, and be aware that some sites treat missing resource requests as a signal. The bandwidth case is in What Is Proxy Bandwidth.

Managing memory across many pages

This is where Puppeteer scripts fail in production. Long runs accumulate memory, and eventually the browser process dies.

• Reuse one page for sequential navigation rather than opening a new page per URL.
• Close pages explicitly when you do open them.
• Restart the browser periodically on long runs, such as every few hundred navigations.
• Cap concurrency. Browser pages are heavy, and a machine that starts swapping produces timeouts that look like proxy failures.
• Watch for orphaned processes. A crash that skips your cleanup leaves browsers running.

The general reliability points are in Monitoring Scraper Health.

Consistency check

A real browser gives you a realistic fingerprint, which helps. Make sure the rest of the session agrees with it: timezone and locale matching the proxy's location, a viewport consistent with the user agent, and no conflicting headers you set manually. Mismatches are a detection signal, as we explain in Why Antidetect Browsers Need Proxies.

When a headless browser is overkill

If the page is server-rendered, use an HTTP client. It is faster, uses a fraction of the memory, scales more cheaply and avoids the whole class of waiting and lifecycle problems above.

Before adopting Puppeteer, check whether the data is available from a JSON endpoint the page calls, which is frequently smaller and more stable. The technique is in Stop Scraping the Page, Find the API Instead.

Puppeteer versus Playwright

Playwright has better waiting semantics, native proxy authentication and cheaper context isolation. Puppeteer is leaner and fits naturally into Node projects. The fuller comparison is in Playwright vs Selenium, and the Playwright proxy setup is in Playwright Scraping with Proxies.

Verify an endpoint before scaling with the proxy checker, which reports the exit IP and added latency.