HTTP 409 Conflict: Causes and Fixes

A 409 means the request conflicts with the current state of the resource. It is an application-level status rather than a network or authentication one, which is why it is less common in scraping than in integrations.

When it appears, it usually tells you something useful about concurrency.

What 409 signals

The server is saying that the request would produce an inconsistent state given what currently exists. Three patterns account for most occurrences.

Duplicate creation. You are trying to create a resource that already exists, typically a unique name, email or identifier.

Concurrent modification. Two clients, or two of your own workers, are attempting to modify the same resource. The second write is rejected rather than silently overwriting.

Optimistic locking. The client supplies a version or If-Match value, and the resource changed since it was read.

The common thread is state. A 409 is about what exists, not about who you are or how fast you are going.

Where it appears in scraping

Scraping is mostly reads, so 409 is uncommon, but it shows up in a few situations:

Authenticated flows with writes. Adding to a cart, saving a search, submitting a form. A retry after a timeout can conflict with a write that actually succeeded, which is the classic case.

Session state mismatches. A stale session token, or a session that a previous request invalidated. This is common where the site rotates a token per action.

Duplicate submissions. A retried form submission that the server treats as a second attempt.

Pagination tokens. Occasionally a cursor that has been superseded returns 409 rather than a new result set.

That first case is the one that causes damage, because a retry of a write that already succeeded can produce a duplicate action.

The retry trap

This is the important section. If you retry indiscriminately on failure, you will eventually resubmit a write that succeeded but whose response you never received.

The failure sequence looks like this:

1. You send a write request.
2. The server processes it successfully.
3. The response is lost: a timeout, a dropped connection, a proxy change.
4. Your client retries.
5. The server sees a duplicate attempt and returns 409, or worse, processes it again.

A 409 here is the polite outcome. The impolite one is a duplicate record.

Idempotency is the fix

Idempotency means a request can be repeated without changing the outcome beyond the first application. Two approaches:

Client-generated idempotency keys. Send a unique key with the request. The server records it and returns the original result if the same key arrives again rather than processing twice. Many APIs support this, and the header name varies.

import uuid

headers["Idempotency-Key"] = str(uuid.uuid4())  # one per logical operation, reused on retry

The key point is that the key is generated once per logical operation and reused across retries. Generating a fresh key per attempt defeats the mechanism entirely.

Conditional requests. Send the version you read, and let the server reject a stale write.

headers["If-Match"] = etag_from_previous_read

On a 412 or 409, re-read the resource, decide whether your write still applies, and try again with the new version. That is the correct loop for optimistic locking.

Retry policy that distinguishes statuses

Not every failure deserves a retry, and grouping them is a common mistake.

Retrying a 409 immediately will produce another 409. Retrying a 422 forever produces a loop. The discipline is in Rate Limiting vs Blocking for the transport side, and this is the application-level counterpart.

Handling it in a pipeline

For scrapers that write, a few practices prevent duplicates:

• Check before write where the cost allows, using a known identifier rather than a fuzzy match.
• Use idempotency keys for any create operation.
• Record the outcome of every write attempt, including the response, so a retry decision can be informed rather than automatic.
• Make deletes idempotent in your own design: deleting an already-deleted item should succeed rather than error, which keeps retries safe.
• Do not retry across a session change. If the session or proxy changed, the write may not be replayable on the new identity. The coherence problem is described in Rotating vs Sticky Proxies.

When 409 is not about you

Occasionally a 409 reflects a shared resource contention you cannot avoid, such as a globally unique name being taken. In that case the fix is to change your input rather than your retry logic, and no amount of backoff will help.

Read the response body where the API provides detail. Many return a machine-readable reason alongside the status, which resolves the ambiguity instantly and saves you from debugging the wrong layer. Log it, along with the request identifier, so the pattern is visible over time, per Monitoring Scraper Health.