Idempotency is easy until the second request is different (opens in new tab)

This is all way too much. If you see a duplicate idempotency key, skip the replay and always return 409. This becomes a client problem. Clients already need to help enforce idempotent contracts; "check for conflict response" is not an onerous imposition.

I've built multiple ecommerce APIs with this approach and they work great. No heroic measures required. You can often satisfy this contract with a unique constraint; if not, a simple presence check in redis. No hashing or worrying about PII.

My rant about this: https://github.com/stickfigure/blog/wiki/How-to-%28and-how-n...

Don't fix other people problems.

If idempotent key was seen then send back response.

Clients intention is outside the scope. If contract says "idempotency on key" the idempotent response on key. If contract says "idempotent on body hash" then response on body hash (which might or might not include extra data).

APIs are contracts. Not the pinky promise of "I'll do my best guess"

This is an excellent article, I’ve seen almost all of the issues it calls out in production for various APIs. I’ll be saving this to share with my team.

I’ve seen two separate engineers implement a “generic idempotent operation” library which used separate transactions to store the idempotency details without realizing the issues it had. That was in an organization of less than 100 engineers less than 5 years apart.

One other thing I would augment this with is Antithesis’ Definite vs Indefinite error definition (https://antithesis.com/docs/resources/reliability_glossary/#...). It helps to classify your failures in this way when considering replay behavior.

Well, it is "reality has a surprising amount of detail"[1] all over again. Or rather a good specific example for it.

[1] http://johnsalvatier.org/blog/2017/reality-has-a-surprising-...

If the second request is different, then idempotency doesn't apply.

A couple of years ago, we experienced a silent data corruption incident in our checkout process due to this specific edge case.

A user would generate the idempotency key by loading the front-end application, adding item(s) to their cart, submitting their order but timing out. The user would then navigate back to the front-end application and add another item and submit the order again. Since the user is submitting an identical idempotency key to the same transaction, our payment gateway would look up the request/transaction by idempotency key and see in its cache that there was a successful (200 OK) response to the previous request. The user now believes they purchased three items, however, our system only charged and shipped on two of the orders.

Consequently, the lesson we take away from the aforementioned incident is idempotency keys are really composite keys (Client_Provided_Key + Hash(Request_Payload)).

If a system receives an identical idempotency key (but with a different request payload) the idempotency key should be rejected with a 409 Conflict response with a message similar to "Idempotency key already used with different request payload". Alternatively, some teams argue it should be returned with a 400 Bad Request response. Systems should never return a failed cache response or replace old entries of data.

This article explains how to unlock your flow. The final idempotent key will not be located until the first request completes, but will rather exist when the request is in progress.

To safely accomplish your goal, you have to follow the following steps:

1. Acquire a distributed lock on the idempotent key.

2. Check for the existence of a key in your persistent store.

3. If an existing key is found, verify the hash of the payload against the hash for the payload type. If the hashes do not match, return a 409 error.

4. If the hashes match, look up the status of the payload. If the status shows COMPLETED in the persistent store, return the cached response. If the status shows PENDING in the persistent store, return a 429 Too Many Requests to the user or hold the connection open until the request reaches a PENDING state.

5. After processing the request, save the response to the persistent store before releasing the lock.

While this may look simple on paper, creating a distributed locking state machine for a single API endpoint is typically how developers have their first aha moments with idempotency. Becoming idempotent is often an enormous architectural shift and not just a middleware header check.

The real problem is that sticking an idempotency key onto an operation doesn’t make it idempotent.

It may improve efficiency where a protocol doesn’t assure exactly-once delivery of messages, but it cannot help you with problems other than deduplication of identical messages.

Creating a payment is not an idempotent operation. If the economics of the operation can differ when the “idempotency” key remains the same then you’ve just created a foot-gun in your API.

You can document that you’re going to ignore “duplicate” requests that share an idempotency key but that’s just user-hostile. The system as a whole is broken as designed.

You, like many clowns in the tech industry do not understand Idempotency.

So "Imma break it down for ya". Since your feeble brain cant do it.

Idempotency literally MEANS: "Empowered by Identity"

IF you actually understood that which you dont, you would know that sending a POST with an ID is a 400 error.

This seems to assume retrying a command should result in the same response, but I am not sure I agree.

Idempotency is about state, not communication. Send the same payment twice and one of them should respond "payment already exists".

i dont disagree with the problem, but this sort of Idempotency-Key header is kind of outsourcing the de-duplication to the client. If the client sends a different request with same Idempotency-Key header its the user's (client's) fault. Its also circumventing the fact that its the effect that should apply to give the same state, you could design the API itself to be idempotent wrt to some other property such as the transaction id. The designs I have seen using an explicit Idempotency-Key header has usually been added on after launch.

> “Put an Idempotency-Key on the request. Store the response. Replay it on retry.”

Is this the new normal? Assert something, that id clearly broken as the correct, then write a blog fixing their broken logic?

You don’t replay it on retry. You signal it is a success on first try, and subsequent request with the same key return 409.

Anything else and you are doing it wrong.

I think this article (and the author's previous articles on their blog) is quite clearly AI written. It has such a frustratingly punctuated cadence and really does not serve the reader anything valuable.

> Maybe it arrives while the first request is still running. Now your idempotency layer is part of your concurrency control.

I recently designed a system where this had to be taken into consideration. I find my solution very elegant: When the request arrives, I put the pending request into a map, keyed by the idempotenceId. This whole operation is executed in one step. Now the event loop may process other requests. If one of them has the same key, it will await the same response object from the store. And then, once i have the response, I resolve both promises with it.

A queue with a pre-flight check can assist with this quite well. Requests are queued and executed ASAP and you use a checker function to verify whether future requests/jobs can run. Just check that idempotency key and if it's already in the queue/db, skip it (and if need be, log the double attempt for future forensics).

You keep the hash of the request so that you can reject a subsequent request with a different body. This has helped me surface bugs and data issues in other systems.

I will add: if you have an operation that adds a record to a database (like a payment in the OP example), don’t forget to have some field that the client specifies that can be used to find and query the status of that record later. This field can be the idempotency key itself or another field.

Or you can completely forget this feature and make it really awkward for the client to reconcile their view of the world with yours and/or to check in the request later. cough Mercury cough.

It is, just barely, acceptable to generate the identifier server side and return it to the client.

I'm not sure this word "idempotent" means what you think it means.

Why does a second request need to replay the response? It seems to solve a ton of issues to just send back "request key already used" regardless of whether the request is the same or whether the previous request succeeded or failed. Is there a common situation where the replay is needed? I think of idempotency being important to deal with caching and the like, not something you should build your client around.

Idempotency means f(x) = f(f(x)).*

Here x is interpreted as state and f an action acting on the state.

State is in practice always subjected to side effects and concurrency. That's why if x is state then f can never be purely idempotent and the term has to be interpreted in a hand-wavy fashion which leads to confusions regarding attempts to handle that mismatch which again leads to rather meandering and confusing and way too long blog posts as the one we are seeing here.

*: I wonder how you can write such a lengthy text and not once even mention this. If you want to understand idempotency in a meaningful way then you have to reduce the scenario to a mathematical function. If you don't then you are left with a fuzzy concept and there isn't much point about philosophizing over just accepting how something is practically implemented; like this idempotency-key.

An idempotency key enables exactly-once semantics in a distributed system.

Half of the mentioned issues are issues of atomicity, not idempotency. If I make a request, and the server crashes midway and doesn't send some crucial events, that's an issue whether or not I send a second request.

From a cursory read, only the part up to "what if the second request comes while the first is running" is an idempotency problem, in which case all subsequent responses need to wait until the first one is generated.

Everything else is an atomicity issue, which is fine, let's just call it what it is.

Idempotency is easy if you don't use mutable state in your middleware.

Auth, logging, and atomicity are all isolated concerns that should not affect the domain specific user contract with your API.

How you handle unique keys is going to vary by domain and tolerance-- and its probably not going to be the same in every table.

It's important to design a database schema that can work independently of your middleware layer.

The article mixes transaction semantics (distributed or otherwise) with idempotency.

Idempotency or not, many points in the articles are are about atomic transactions.

Just do what azure’s api is doing. Just return http/500 on all valid calls. Problem solved.

The point of idempotency is safe retries. Systems are completely fallible, all the way down to the network cables.

The user wants something + the system might fail = the user must be able to try again.

If the system does not try again, but instead parrots the text of the previous failure, why bother? You didn't build reliability into the system, you built a deliberately stale cache.

This article seems to be missing an example use-case for this functionality, it is very unclear to me that this is a good idea. Your job is hopefully to build an API with the simplest and most reliable contract to meet the needs of the client. Sometimes that involves saying, "human technology does not have a reliable way to support this expectation, but here's what else could work."

"An operation is idempotent if applying it once or many times has the same intended effect."

No -- Idempotent means _no_ side-effects.

Why isn't github in the list of self-hosted options?

> Maybe the first request called a payment provider, the provider accepted it, and your process died before recording the result. Now your database cannot infer whether money moved.

This entire example is bad design. It's bad, bad design. I'm sorry, but if this is your example, you are doing it wrong in every way. There are ways to handle these sorts of things, well-known and well-established patterns. You are using none of these here.

I get it, it's an example, but it's a poor example. You should change it before someone assumes what you are talking about is sensible or reasonable in a production environment. Or at least put a warning.

Idempotency is just the next generation realizing that goto and global variables are still antipatterns.

And then some lazy birdbrain will come up with some new way to either jump to a random place in the code without guardrails on program state, or referencing data that other code or threads could have touched, and they'll call it a time saving feature.

And then we will all learn the hard way that those annoying restrictions were in place for a reason.

This is the great circle of life and death and rebirth

yes I always thought it's an easy thing. but I changed my mind recently when I had to deal with it.

A lot little things you need to think of. For example.

Client sends a request. The database is temporarily down. The server catches the exception and records the key status as FAILED. The client retries the request (as they should for a 500 error). The server sees the key exists with status FAILED and returns the error again-forever. Effectively "burned" the key on a transient error.

others like:

- you may have Namespace Collisions for users... (data leaks) - when not using transactions only redis locking you have different set of problem - the client needs to be implmented correctly. Like client sees timout and generates a new key, and exactly once processing is broken - you may have race conditions with resource deletes - using UUID vs keys build from object attributes (different set of issues)

I mean the list can get very long with little details..

I really hate the POST verb for RESTish APIs because it cannot be idempotent without implementing an idempotency layer. Other verbs are naturally idempotent. Has anyone tried foregoing POST routes entirely? Theoretically you can let the client generate an ID and have it request a PUT route to create new entities. This would give you a tiny amount of extra complexity on the client, but make the server simpler as a trade-off.

Sigh. What this article is badly saying is that they really don't understand the difference between a *transaction* versus idempotency.

You want a rebuildable environment after testing blows it up? Idempotent build scripts.

You want to sell crap from a web interface? Thats a transaction. If you do 'repeat a sale', thats a new transaction, with new goods, with newer date.

Forcing 1 paradigm on a different one always results in gnashing of teeth and sadness. But I guess it gets the blog hits for that dopamine rush.

Um, is the headline a joke?

Like, I thought the entire definition had to do with "the exact same thing twice."

skill issue lol, it's not idempotent anymore, same key for different requests? Heard of a nonce?

> If you’re still in school, here’s a fact: you will learn as much or more every year of your professional life than you learned during an entire university degree—assuming you have a real engineering job.

This rubs me the wrong way. It's stated as fact without any trace of evidence, it is probably false, and it seems to serve no purpose but to make struggling students feel worse (and make the author feel superior).