Running Scrapers and Proxies in Docker

Docker solves a real problem for scrapers: the environment. Browser versions, system libraries and dependencies differ between machines, and a scraper that works on one laptop fails on another for reasons unrelated to the code.

Containers fix that. They also introduce a handful of specific issues worth knowing before you spend an afternoon debugging them.

A minimal scraper image

FROM python:3.12-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY . .

CMD ["python", "collect.py"]

That is enough for an HTTP-based scraper. Two habits keep the image small and the build fast: copy requirements.txt before the source so the dependency layer caches, and use --no-cache-dir to avoid leaving pip's download cache in the layer.

Adding a browser

Headless Chrome needs system libraries that the slim image does not include, and this is where most container builds go wrong.

FROM python:3.12-slim

RUN apt-get update && apt-get install -y --no-install-recommends \
    ca-certificates fonts-liberation libglib2.0-0 libnss3 libx11-6 \
    libxcomposite1 libxdamage1 libxrandr2 libgbm1 libasound2 \
    && rm -rf /var/lib/apt/lists/*

RUN pip install --no-cache-dir playwright && playwright install --with-deps chromium

The playwright install --with-deps step installs the browser and its dependencies, which is the reliable path. Hand-assembling the library list is error-prone because the requirements change with Chrome versions. The trade-off is image size, which grows substantially with a browser included. That is a real cost in CI, where images are pulled on every run.

The shared memory crash

The issue that wastes the most time: Chrome uses /dev/shm for shared memory, and Docker's default of 64 MB is too small for a browser doing real work. The symptoms are cryptic crashes or pages failing to render, with no error pointing at memory.

Two fixes:

docker run --shm-size=1gb my-scraper

Or, in a compose file:

services:
  scraper:
    build: .
    shm_size: "1gb"

Alternatively, pass --disable-dev-shm-usage as a browser argument so Chrome writes to disk instead. That works but is slower, so the shared-memory increase is preferable where you control the run configuration.

Passing proxy credentials

Environment variables are the right mechanism, and they keep secrets out of the image entirely.

docker run --env-file .env my-scraper

import os
import requests

proxy = os.environ["LB_PROXY_URL"]
session = requests.Session()
session.proxies = {"http": proxy, "https": proxy}

Three cautions. Never bake credentials into the image, because layers are inspectable. Be careful with build arguments, which are recorded in image history. And remember that anyone with access to the running container can read its environment, so the usual credential hygiene in Proxy Authentication still applies.

Networking considerations

The proxy must be reachable from inside the container. A proxy on the host machine is not at localhost from the container's perspective; it is at the host's gateway address, or reachable by hostname. This trips people up consistently.

DNS resolution inside containers can differ from the host, particularly with custom DNS configurations. If a target resolves differently, that is the reason.

Verify the exit address from inside the container, not from your host. The two may not match if the configuration was not applied where you expected. The proxy checker works from inside a container as well as outside, and it is worth running in both places.

A compose setup for a browser pool

For a scraper that needs several browser workers, a compose service with a concurrency limit is a reasonable middle ground.

services:
  scraper:
    build: .
    shm_size: "1gb"
    env_file: .env
    deploy:
      replicas: 3
    restart: unless-stopped

Three replicas each with one worker, one session and one endpoint is a better shape than one container running three browsers, because the per-endpoint concurrency rule is easier to enforce. The principle is in What Is a Proxy Pool.

Operational habits

Log to stdout rather than to files, so the container platform captures it and you do not need volume mounts for logs.

Exit non-zero on failure, so an orchestrator can restart or alert. A scraper that swallows an exception and exits zero is invisible.

Write outputs outside the container. A database, object storage or a mounted volume. Container filesystems are ephemeral, and losing a completed run because it wrote to /app/output is a common and avoidable mistake.

Set memory limits deliberately. Browsers are hungry, and an unbounded container will eventually take down its host. Note the lifecycle concerns in Puppeteer Web Scraping.

Pin image versions. python:3.12-slim moves, and a base-image change can break a working build months later.

When Docker is worth it

Yes, when your scraper needs a browser, when it runs on multiple machines, when you want reproducibility, or when you deploy it alongside other services.

Less so, when you are running a simple scheduled script and CI already provides the environment. In that case the container is extra machinery for no gain, and the approach in Scheduling Scrapers with CI is simpler.

The proxy layer is unaffected by the packaging choice, and the selection logic is in Choosing a Proxy for Web Scraping.