Scheduling Scrapers with CI

A scheduled CI job is the lowest-effort way to run a scraper regularly. You get a scheduler, a runner, secret management and logs without building any of it.

It is also the wrong tool for some jobs, and knowing which is which saves a lot of frustration.

What CI gives you for free

A scheduler. Cron syntax, which most teams already know.

A runner. A machine that starts, runs your job and stops. No server to maintain.

Secret storage. Encrypted environment variables injected at run time, which solves the credential handling problem described in Proxy Authentication.

Logs and history. Every run is recorded, which is the basis of the monitoring in Monitoring Scraper Health.

Artifacts. A place to store outputs, with retention policies.

For a daily or hourly collection job of modest size, that is a complete platform.

A minimal workflow

name: collect

on:
  schedule:
    - cron: "0 */6 * * *"   # every six hours
  workflow_dispatch:         # allow a manual run

jobs:
  collect:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install -r requirements.txt
      - run: python collect.py
        env:
          LB_PROXY_URL: ${{ secrets.LB_PROXY_URL }}
          TARGET_BASE_URL: ${{ secrets.TARGET_BASE_URL }}
      - uses: actions/upload-artifact@v4
        with:
          name: output
          path: output/
          retention-days: 14

Three details worth noting. workflow_dispatch lets you trigger a run manually, which you will want constantly during development. Secrets arrive as environment variables so they never appear in the repository. And the artifact upload gives you a record of what each run produced.

Scheduling caveats

Cron in CI is not precise. Scheduled runs can be delayed under load, sometimes by many minutes. Do not build a workflow that depends on running at exactly 09:00.

Scheduled workflows can be disabled after inactivity. Check the platform's policy, because a repository with no activity for a period may stop running schedules silently. That is a quiet failure mode worth guarding against.

Timezone matters. Cron expressions are usually UTC. A "daily at 09:00" job needs converting for your target market, and get it wrong and you will be collecting at an odd local hour. Our note on UTC handling in Residential Proxies for Travel Fare Aggregation applies to scheduling too.

Concurrent runs can overlap. If a job takes longer than its interval, two runs may execute at once and both draw on your proxy budget. Add a concurrency guard.

The egress IP problem

This is the reason proxies are usually necessary with CI, and it is easy to overlook.

CI runners use cloud provider addresses. Those are datacenter ranges, and defended targets classify them immediately. A job that works perfectly on a developer's laptop may be blocked on the runner, because the network identity changed.

Two consequences:

• Test on the runner early, not after you have built everything.
• Plan for a proxy, and choose the type based on the target per Choosing a Proxy for Web Scraping.

Configuration is the same as anywhere else, with the proxy URL in a secret:

import os
import requests

proxy = os.environ["LB_PROXY_URL"]
session = requests.Session()
session.proxies = {"http": proxy, "https": proxy}

Verify the actual exit from the runner rather than assuming. A silent bypass looks identical to success, and the proxy checker confirms the exit IP and added latency.

Storing outputs

Three options, in increasing order of suitability:

Artifacts. Good for short-lived results and for debugging a run. Retention policies usually expire them, so they are not a system of record.

A database. The right answer for anything that matters. History lets you distinguish real change from a scrape glitch, which is the argument in Residential Proxies for Price Monitoring.

Object storage. Suitable for raw page archives or large exports that a later process consumes.

A common and sensible pattern is to write the structured result to a database and archive raw responses to object storage, so a parser fix can be re-run without re-fetching.

Failure handling

A scheduled job that fails silently is worse than one that does not run, because you believe the data is current.

• Fail the job on a collection failure rather than exiting zero. Most platforms notify on failure by default.
• Add a sanity check. If the run produced zero records, that is a failure regardless of exit code, which is the coverage check from Handling Pagination in Scrapers.
• Alert on trends, not just failures. A gradual drop in records per run is an early signal of parser drift.
• Keep logs long enough to compare a bad run with a good one.

When CI is the wrong tool

Long-running jobs. Runners have time limits, commonly six hours. A crawl that takes longer needs a real server or a container job.

High-volume jobs. Runner minutes are metered, and a large crawl is expensive compared with a small always-on instance.

Jobs needing a persistent process. A browser pool or a connection kept open does not fit the run-and-stop model. See What Is a Proxy Pool.

Jobs requiring a fixed IP. Runners are ephemeral, so allowlisting is impractical. That is what the proxy solves, but it means the proxy is mandatory rather than optional.

Real-time requirements. CI scheduling is measured in minutes, not seconds.

A sensible division

Use CI for jobs that are short, periodic and modest in volume: daily price checks, sitemap monitoring, a handful of API pulls. Use a server or container for anything long-running, high-volume or needing persistent state.

The proxy layer is identical in both cases, and the sizing method is in How Much Residential Bandwidth Do You Need.