Data Collection Ethics for Engineering Teams

Ethics in collection work is usually presented as a legal question, and the legal answer is necessary but insufficient. Two systems with identical legal exposure can differ enormously in whether they should exist, and engineering teams are the ones who decide the details that make the difference.

This is a practical framework, not legal advice. The legal background is in Is Web Scraping Legal?.

Start with what you are collecting

The first distinction that matters is between facts about things and data about people.

Non-personal data. Product prices, stock levels, listing counts, page structure, publicly published statistics. Low individual impact, and the main questions are commercial and reputational.

Personal data. Names, contact details, photos of identifiable people, profile information, anything linked to an identifiable individual. Brings obligations: a lawful basis, a retention limit, and rights for the subjects under regimes like GDPR.

Special category data. Health, political opinions, sexual orientation, biometric data. Highest obligation, and collection should require a specific and compelling justification.

Most collection projects mix these without noticing. Scraping a marketplace for prices also captures seller names. Collecting classified ads captures phone numbers. The ethical work starts with separating what you need from what arrives attached.

Proportionality

The test: is the collection proportionate to the purpose, and is there a less intrusive way to achieve it?

Three questions to answer explicitly:

What decision will this data inform? If there is no decision, the collection has no purpose and should not run.

Is there a less intrusive route? An official API, a published dataset, a licensed feed, a paid data provider. Where one exists and covers the need, it is almost always the better answer.

Is the scope minimal? Collecting a whole catalogue to answer a question about ten products is disproportionate, and it multiplies both the request volume and the personal data captured.

Proportionality is the principle that most often changes a design, and it is the one teams skip.

Terms, robots and technical signals

These are not the same thing, and they are weighted differently.

Terms of service. A contract. Breaching it is the most common source of actual legal exposure, and it is the reason a technically accessible site may still be off limits.

robots.txt. A technical convention, not law, and not a contract in most jurisdictions. It signals the publisher's intent. Respecting it for the paths relevant to your collection is a reasonable baseline, and ignoring it entirely is a signal about how you operate.

Rate and load. Whether your collection degrades the service for its actual users. This is the harm most often caused by collection and the one least discussed.

Authentication walls. Anything behind a login you are not entitled to use is the clearest line in the whole subject.

A practical rule: if the answer depends on a distinction a non-lawyer would find unreasonable, do not build it.

Retention

Data you have collected is an ongoing obligation, not a one-time decision.

Four policies worth setting before launch:

• A retention period per data class. Raw HTML shorter than parsed records, personal data shorter than non-personal.
• Deletion that actually runs. A policy without an automated job is a policy that does not exist.
• No indefinite raw storage. Keeping every response "in case it is useful" is how teams end up holding personal data they cannot justify.
• Documentation of what is held and why, so the answer to a subject access request is available rather than reconstructed.

Documenting your basis

The artefact that makes all of this defensible is a short written record, written before the collection starts.

For each collection task:

A page per task. It takes an hour and it answers most questions that arise later, from a client, a platform or a regulator.

Pre-launch checklist

Before a collector runs in production:

• Purpose is stated and a decision depends on the data.
• A less intrusive alternative was considered and its rejection is recorded.
• The source's terms were read, and the collection is consistent with them.
• robots directives were checked for the relevant paths.
• Request rates stay within what the site tolerates for human users.
• Personal data fields are identified, and each is either needed or excluded.
• Retention is set per data class, with an automated deletion job.
• Raw responses have a shorter retention than parsed records.
• Access controls limit who can query the collected data.
• The basis document exists and has an owner.
• The collection does not degrade the source for its users.

Where infrastructure sits

The address layer has an ethical dimension of its own, and it is about honesty rather than concealment.

Using a geo-matched address so that a measurement reflects a market accurately is a correctness measure, and it is the same requirement as in E-Commerce Data Scraping. Using one to make traffic unidentifiable is a different thing, and the distinction should be clear in the design document, because it is the first question a reviewer will ask.

Similarly, distributing load across addresses to stay within what a site tolerates is responsible. Distributing load to exceed what the site tolerates without being noticed is not, and the two look identical in code.

For the legal position, see Is Web Scraping Legal?. For the operational side, Proxy Troubleshooting Checklist, and for the address products, Residential and Mobile.