Idempotency Keys Are Easy — Until the Second Request Is Different

A blog post titled "Idempotency Is Easy Until the Second Request Is Different" climbed the Hacker News front page this week, and it struck a nerve with a lot of backend engineers — me included. The thesis is deceptively simple: the textbook idempotency pattern (attach an Idempotency-Key, store the response, replay it on retry) handles the boring case perfectly. The hard part starts when the second request carrying the same key isn't a clean replay of the first one. Same key, different amount. Now what?

I've shipped payment and ordering APIs for e-commerce clients where this exact ambiguity was the difference between "we processed the charge once" and "we double-billed a customer and spent the afternoon writing apology emails." So I want to take the article's argument seriously and show what a correct implementation looks like in .NET — not the happy-path demo, but the version that survives a broken client.

What idempotency actually buys you

Idempotency means an operation can be applied multiple times without changing the result beyond the first application. GET, PUT and DELETE are idempotent by their HTTP semantics. POST is not, and that's where the pain lives: creating an order, charging a card, sending money. These are the operations a client most wants to retry safely, because networks fail in the worst way possible — the request succeeds on the server but the response never makes it back. The client has no idea whether the charge happened. It retries. Without idempotency, you charge twice.

The fix is to let the client supply a unique key per logical operation:

POST /api/payments HTTP/1.1
Idempotency-Key: 2f1c9b7e-3a44-4c2e-9d6e-7b0a8e5f1c23
Content-Type: application/json

{ "orderId": "ord_8842", "amountCents": 1000, "currency": "EUR" }

If the server has already seen that key, it returns the stored response instead of executing the operation again. Stripe popularized this design and it's now the de facto standard. The straightforward version in ASP.NET Core looks like this:

public sealed record IdempotencyRecord(
    string Key,
    string RequestHash,
    int StatusCode,
    string ResponseBody,
    DateTimeOffset CreatedAt);

public interface IIdempotencyStore
{
    Task<IdempotencyRecord?> GetAsync(string tenantId, string key, CancellationToken ct);
    Task<bool> TryReserveAsync(string tenantId, string key, string requestHash, CancellationToken ct);
    Task CompleteAsync(string tenantId, string key, int statusCode, string body, CancellationToken ct);
}

So far, so easy. The article's point — and the part most tutorials skip — is everything below.

The second request is where it gets interesting

Here is the scenario the HN discussion kept circling back to. A client sends a payment with key abc-123 for €10. Then a second request arrives with the same key abc-123, but the body now says €10,000. What should the server do?

There are three plausible interpretations, and they are mutually exclusive:

1. It's a retry. Replay the stored €10 response and ignore the new body.
2. It's a client bug. The client reused a key it shouldn't have. Reject it loudly.
3. It's a new operation. Treat (key + content) as a fresh identity and process €10,000.

The article argues — correctly, in my experience — that for side-effecting APIs the bias should be: same scoped key + a different canonical command is a hard error. Option 2. You return 409 Conflict (or 422), you do nothing, and you make the client's mistake visible immediately.

Why is that the right default? Because option 1 silently discards a request the client believes it sent, and option 3 silently executes an operation the client may never have intended at that amount. Both are catastrophic for money. A client that believes it is safely retrying a €10 payment should never have the server quietly reinterpret the second request as something else. The whole promise of idempotency is "you can retry without thinking." Breaking that promise silently is worse than not offering it at all.

Canonicalising the request

To detect "same key, different request," you need a deterministic fingerprint of the request. Naively hashing the raw bytes is fragile: a reordered JSON property or a whitespace difference would flip the hash and turn a legitimate retry into a false conflict. So canonicalise first.

using System.Security.Cryptography;
using System.Text;
using System.Text.Json;
using System.Text.Json.Nodes;

public static class RequestCanonicalizer
{
    // Produce a stable hash over the semantically meaningful request content.
    public static string ComputeHash(string rawJson)
    {
        var node = JsonNode.Parse(rawJson) ?? throw new JsonException("Empty body");
        var canonical = Canonicalize(node);
        var bytes = Encoding.UTF8.GetBytes(canonical.ToJsonString());
        return Convert.ToHexString(SHA256.HashData(bytes));
    }

    private static JsonNode Canonicalize(JsonNode node) => node switch
    {
        JsonObject obj => new JsonObject(
            obj.OrderBy(p => p.Key, StringComparer.Ordinal)
               .Select(p => KeyValuePair.Create(
                   p.Key,
                   p.Value is null ? null : Canonicalize(p.Value)))),
        JsonArray arr => new JsonArray(
            arr.Select(e => e is null ? null : Canonicalize(e)).ToArray()),
        _ => JsonNode.Parse(node.ToJsonString())!
    };
}

Now { "amountCents": 1000, "currency": "EUR" } and { "currency": "EUR", "amountCents": 1000 } produce the same hash, but changing the amount to 1000000 produces a different one. That's the distinction the conflict check hinges on.

A caveat the article implies but is worth stating: be deliberate about what goes into the canonical command. A trace ID or a client-side timestamp header should not be part of the fingerprint, or every retry will look different. Hash the business intent — the amount, the currency, the target account — and nothing else.

Putting it together in middleware

I prefer to handle idempotency in middleware rather than scattering it across controllers. It keeps the policy in one place and out of the business logic. The flow has three states: never seen the key (reserve and execute), seen it and completed (replay), seen it but the request differs (conflict).

public sealed class IdempotencyMiddleware(RequestDelegate next, IIdempotencyStore store)
{
    public async Task InvokeAsync(HttpContext context)
    {
        // Only guard unsafe methods that opt in via the header.
        if (!HttpMethods.IsPost(context.Request.Method) ||
            !context.Request.Headers.TryGetValue("Idempotency-Key", out var keyValues))
        {
            await next(context);
            return;
        }

        var key = keyValues.ToString();
        var tenantId = context.User.FindFirst("tenant_id")?.Value ?? "anonymous";
        var ct = context.RequestAborted;

        context.Request.EnableBuffering();
        using var reader = new StreamReader(context.Request.Body, leaveOpen: true);
        var rawBody = await reader.ReadToEndAsync(ct);
        context.Request.Body.Position = 0;
        var requestHash = RequestCanonicalizer.ComputeHash(rawBody);

        var existing = await store.GetAsync(tenantId, key, ct);
        if (existing is not null)
        {
            if (existing.RequestHash != requestHash)
            {
                context.Response.StatusCode = StatusCodes.Status409Conflict;
                await context.Response.WriteAsJsonAsync(new
                {
                    error = "idempotency_key_reuse",
                    message = "This Idempotency-Key was already used with a different request body."
                }, ct);
                return;
            }

            // Genuine retry: replay the stored response verbatim.
            context.Response.StatusCode = existing.StatusCode;
            context.Response.ContentType = "application/json";
            await context.Response.WriteAsync(existing.ResponseBody, ct);
            return;
        }

        if (!await store.TryReserveAsync(tenantId, key, requestHash, ct))
        {
            // Another request with the same key is in flight right now.
            context.Response.StatusCode = StatusCodes.Status409Conflict;
            await context.Response.WriteAsJsonAsync(new
            {
                error = "request_in_progress",
                message = "A request with this Idempotency-Key is already being processed."
            }, ct);
            return;
        }

        // Capture the downstream response so we can persist it.
        var originalBody = context.Response.Body;
        using var buffer = new MemoryStream();
        context.Response.Body = buffer;

        await next(context);

        buffer.Position = 0;
        var responseBody = await new StreamReader(buffer).ReadToEndAsync(ct);
        buffer.Position = 0;
        await buffer.CopyToAsync(originalBody, ct);
        context.Response.Body = originalBody;

        if (context.Response.StatusCode is >= 200 and < 300)
            await store.CompleteAsync(tenantId, key, context.Response.StatusCode, responseBody, ct);
    }
}

Two details that separate a toy from a production implementation. First, I only persist successful responses. If the operation failed with a 500, the client should be able to retry it; caching a failure forever would be a trap. Second, the TryReserveAsync reservation handles concurrent duplicates — two retries racing each other before the first one finished. Without it, both pass the "have I seen this key?" check and you're back to double-charging.

Scope the key — it is not globally unique

The single most important design decision in the article is easy to miss: the idempotency key is not globally unique unless you deliberately make it so, and usually it should not be. A broken client generating abc-123 for everything should only ever collide with itself, not with another tenant who happened to pick the same string. That's why every method on my store takes a tenantId. The uniqueness constraint lives on the composite, not the key alone.

In EF Core that's a one-liner, and it's the line that keeps tenant A from replaying tenant B's response:

modelBuilder.Entity<IdempotencyRecord>()
    .HasKey(r => new { r.TenantId, r.Key });

The atomic reservation maps cleanly onto the database. With PostgreSQL, an INSERT ... ON CONFLICT DO NOTHING gives you a lock-free reserve-or-fail in a single round trip:

public async Task<bool> TryReserveAsync(
    string tenantId, string key, string requestHash, CancellationToken ct)
{
    const string sql = """
        INSERT INTO idempotency_records (tenant_id, key, request_hash, status_code, response_body, created_at)
        VALUES ({0}, {1}, {2}, 0, '', now())
        ON CONFLICT (tenant_id, key) DO NOTHING;
        """;
    var rows = await _db.Database.ExecuteSqlRawAsync(sql, ct,
        tenantId, key, requestHash);
    return rows == 1; // 1 = we won the race, 0 = someone already reserved it
}

For high-throughput endpoints I reach for Redis instead, using SET key value NX EX 86400 for the reservation and a TTL so stale in-flight records expire. The principle is identical — atomic reserve, scoped key, bounded lifetime — only the storage engine changes.

Where I'd push back

The article's "hard error on mismatch" rule is the right default, but it isn't universal. For some APIs the key is genuinely a deduplication window, not a contract about content, and treating a differing body as a new operation is the documented behaviour. The discipline is to decide explicitly and document it, rather than letting whatever your framework does by accident become your semantics. The failure mode I've cleaned up after is never "the team chose option 3 deliberately." It's "nobody chose anything, and the behaviour emerged from a GetOrCreate call somebody wrote in a hurry."

There's also the question of how long you retain keys. Stripe keeps them for 24 hours; that's a reasonable default. Keep them forever and your store grows without bound; expire them too fast and a client retrying after a long outage gets a fresh execution instead of a replay. Tie the TTL to how long your clients realistically retry, and make it longer than your longest backoff schedule.

Takeaway

Idempotency keys are one of those features that look finished after twenty minutes and reveal their real depth only in production, usually at the worst possible time. The trending HN piece is right that the easy case is a distraction. The work is in the second request: canonicalise the body so genuine retries are recognised, reject mismatched reuse with a 409 so client bugs surface early, scope the key per tenant so it can't collide across boundaries, and reserve atomically so concurrent retries don't slip through. Do those four things and you've built something that actually keeps its promise — that a client can retry without thinking, and your customers get charged exactly once.

---

Building a payment or ordering API and want a second pair of eyes on the idempotency and webhook handling? Get in touch — it's most of what I do.

Source: "Idempotency Is Easy Until the Second Request Is Different" and the Hacker News discussion.