The valley of webhooks: what to build when delivery is at-least-once

There is a post climbing Hacker News this week with a title that stuck with me: "The Valley of Webhooks." The author has built webhook-based data synchronisation three separate times, and the piece makes a calm, careful case that webhooks are a local optimum. A comfortable dip in the map that turns out to be hard to climb out of, because every problem they cause already has a workaround, and every workaround already has a vendor.

I have written that same code. So has most of the industry. I think the diagnosis is correct, which is exactly why it is worth being precise about what you do with it on a Tuesday, when the ticket says "our copy of the customer's subscription is wrong again."

The diagnosis is right

The core argument splits webhooks into two jobs. Job one is triggering a side effect: send a receipt, kick off a CI build, post to Slack. Job two is replicating a provider's data into your own database so you can query it locally. Webhooks were born for job one. Most of us reach for them for job two, and job two needs exactly the properties webhooks lack: ordering, completeness, a way to bootstrap from the current state, and a way to verify you got everything.

The author is blunt about where this ends. You write a nightly reconciliation cron that re-pulls everything and overwrites your copy. In the post's words, that cron is "a written confession. It says: I do not trust the copy I built, and I have no way to know when it's wrong, so I will re-derive it from scratch every night, forever." The example that made me wince: a customer had cancelled months earlier and the database still said active, because some customer.subscription.deleted "had evaporated between Stripe and us, and nothing anywhere was capable of noticing."

The missed event is almost forgivable. What should scare you is that nothing noticed.

Most of us are on the receiving end

Here is where I part ways with how the article usually gets shared. The proposed fix, which the author sketches as SCROLL (Synchronized Change Replication Over Line Logs), asks the provider to expose an ordered, cursor-addressed log you pull from. It is a good design. I will build the provider side of it below. But you do not get to redesign Stripe's API this quarter. On the day the ticket lands, you are the consumer, and your job is to build a receiver that stays correct on top of delivery you cannot change.

So let me give the consumer playbook first, in .NET, the way I actually write it.

A receiver that survives at-least-once

Four rules. None of them clever. All of them boring on purpose.

Verify the signature over the raw body

Check the signature over the exact bytes, before you deserialize anything, with a constant-time comparison. A timing-safe compare matters because naive string equality leaks how many leading bytes matched.

using System.Security.Cryptography;
using System.Text;

// Header shape "sha256=<hex>", e.g. GitHub's X-Hub-Signature-256.
static bool HasValidSignature(byte[] rawBody, string secret, string? header)
{
    const string prefix = "sha256=";
    if (header is null || !header.StartsWith(prefix, StringComparison.Ordinal))
        return false;

    byte[] provided;
    try { provided = Convert.FromHexString(header.AsSpan(prefix.Length)); }
    catch (FormatException) { return false; }

    using var hmac = new HMACSHA256(Encoding.UTF8.GetBytes(secret));
    var computed = hmac.ComputeHash(rawBody);

    // Returns false on any length mismatch, constant time otherwise.
    return CryptographicOperations.FixedTimeEquals(computed, provided);
}

The word "raw" is load-bearing. If you bind the request to a model and re-serialize it to check the signature, you will fail on the day the provider adds a field or changes some whitespace. Read the bytes off the wire and hash those.

app.MapPost("/webhooks/billing", async (HttpRequest req, Inbox inbox) =>
{
    using var buffer = new MemoryStream();
    await req.Body.CopyToAsync(buffer);
    var body = buffer.ToArray();

    var header = req.Headers["X-Signature-256"].ToString();
    if (!HasValidSignature(body, BillingSecret, header))
        return Results.Unauthorized();

    var evt = JsonSerializer.Deserialize<ProviderEvent>(body)!;
    await inbox.StoreAsync(evt.Id, evt.Type, body);

    return Results.Ok();
});

Acknowledge in milliseconds, process out of band

Notice what that handler does not do: it does not run business logic. It writes the event to an inbox table and returns 200. If you do the real work inline, a slow downstream call turns into a timeout, the provider retries, and now you are processing the same event twice while the first pass is still running. Store, acknowledge, then let a background worker drain the inbox. The provider's retry timer stops being your problem the moment the row is safe on disk.

Assume every event arrives at least twice

At-least-once is a promise, and the promise is duplicates. So the write has to be idempotent, keyed on the provider's own event id. Let the database enforce it with a unique index and treat the collision as a success.

public sealed class InboxEvent
{
    public long Id { get; set; }
    public required string EventId { get; init; }   // provider's id, this is the dedup key
    public required string Type { get; init; }
    public required byte[] Payload { get; init; }
    public DateTimeOffset ReceivedAt { get; init; } = DateTimeOffset.UtcNow;
    public bool Processed { get; set; }
}

// modelBuilder.Entity<InboxEvent>().HasIndex(e => e.EventId).IsUnique();

public sealed class Inbox(AppDb db)
{
    // Returns true if the event was new, false if we had already stored it.
    public async Task<bool> StoreAsync(string eventId, string type, byte[] payload)
    {
        db.Add(new InboxEvent { EventId = eventId, Type = type, Payload = payload });
        try
        {
            await db.SaveChangesAsync();
            return true;
        }
        catch (DbUpdateException ex) when (ex.InnerException is PostgresException { SqlState: "23505" })
        {
            db.ChangeTracker.Clear();   // drop the failed insert so the context stays usable
            return false;               // 23505 is unique_violation: we have seen this event
        }
    }
}

On SQL Server the guard is error number 2627 or 2601 instead of Postgres SQLSTATE 23505. Same idea, different constant.

Do not trust the order

Events arrive out of order. A membership.created can land before its user.created. If your processing assumes the parent exists, it throws, the provider retries, and you get lucky or you do not. Two honest options. Either process each event as an upsert keyed on the resource id, so a later state simply overwrites an earlier one, or, when you genuinely need ordering, buffer the event and wait for its prerequisite instead of failing hard. What you never do is assume the sequence is right because it was right in testing.

Reconciliation is fine, as long as it is loud

The article frames the nightly reconcile as a confession. I read it differently. Reconciliation is a decent safety net. A silent one is the actual sin. If your cron quietly overwrites active with canceled every night and never tells anyone, you have hidden a data-integrity bug behind a scheduled job, and you will hear about it from a customer instead of from your dashboard.

So reconcile by pulling the provider's event list, run it through the same idempotent inbox, and count the drift.

public async Task<int> ReconcileAsync(CancellationToken ct)
{
    var cursor = await _state.GetLastEventIdAsync();
    var options = new EventListOptions { Limit = 100 };
    if (cursor is not null) options.StartingAfter = cursor;

    var events = new EventService();
    var repaired = 0;

    await foreach (var evt in events.ListAutoPagingAsync(options, cancellationToken: ct))
    {
        var isNew = await _inbox.StoreAsync(evt.Id, evt.Type, Encoding.UTF8.GetBytes(evt.ToJson()));
        if (isNew) repaired++;                 // a gap the webhook never delivered
        await _state.SaveLastEventIdAsync(evt.Id);
    }

    if (repaired > 0)
        _logger.LogWarning("Reconcile healed {Count} missing events", repaired);

    return repaired;
}

That repaired counter is the whole point. Wire it to an alert. If reconciliation is routinely fixing dozens of events, your webhook path has a real problem and you now have the number to prove it. If it fixes zero for a month, you get to trust your live path a little more. And because it is a scheduled job doing quiet, important work, monitor the job itself. A reconcile that silently stops running is how you end up back at a customer telling you their subscription is wrong.

Stripe helps here on purpose. It keeps thirty days of events and exposes /v1/events for exactly this kind of listing. WorkOS ships a cursor-paginated Events API for the same reason. The escape hatch already exists in the platforms that thought about it.

If you own the API, ship a log

Now the provider side, because sometimes you are the one sending the webhooks. This is where the SCROLL argument earns its keep. Do not make your consumers build everything above. Give them an ordered log to pull.

The trick that makes it correct is the transactional outbox. Write the change record in the same database transaction as the entity it describes. If they commit together, the log can never disagree with the data.

public async Task CancelAsync(string subscriptionId)
{
    var sub = await _db.Subscriptions.SingleAsync(s => s.Id == subscriptionId);
    sub.Status = "canceled";

    _db.Changes.Add(new Change
    {
        Resource = "subscription",
        ResourceId = sub.Id,
        Type = "deleted",
        Deleted = true,                        // a tombstone: the delete stays visible
        Data = JsonSerializer.Serialize(sub)
    });

    // One SaveChanges, one transaction: the log and the row commit together.
    await _db.SaveChangesAsync();
}

Change.Seq is a monotonic, database-assigned key. That is your cursor. The endpoint is a single GET that streams newline-delimited JSON from a given cursor forward.

app.MapGet("/v1/changes", async (long? since, int? limit, AppDb db, HttpResponse res) =>
{
    var after = since ?? 0;                     // no cursor means bootstrap from zero
    var take = Math.Clamp(limit ?? 100, 1, 1000);

    var page = await db.Changes
        .Where(c => c.Seq > after)
        .OrderBy(c => c.Seq)
        .Take(take)
        .ToListAsync();

    res.ContentType = "application/x-ndjson";
    await using var writer = new StreamWriter(res.Body, Encoding.UTF8);
    foreach (var c in page)
        await writer.WriteLineAsync(JsonSerializer.Serialize(new
        {
            seq = c.Seq, resource = c.Resource, id = c.ResourceId,
            type = c.Type, deleted = c.Deleted, data = c.Data
        }));
});

Look at what the consumer gets for free. Order, because the rows come out by Seq. Bootstrap, because since=0 reads from the beginning of time. Deletes, because a tombstone is a real event rather than a row that quietly vanished. Resumability, because they store the last seq and ask again from there. No endpoint to register, no signing secret to rotate, no public URL that has to reach a laptop behind a NAT. Every change arrives over a connection they opened, with the API key they already have.

When webhooks are still the right tool

None of this means webhooks are wrong. For job one they are great. A "payment succeeded, send the email" trigger, where a rare miss gets caught by a retry, does not need a log and a cursor and an outbox table. Push latency is a genuine feature: the event shows up in a second instead of on your next poll. The failure mode of "we sent the welcome email twice" is a shrug, not an incident. Keep webhooks for the side effects they were designed to fire. Reach for a log the moment the correctness of a replicated dataset is on the line.

Frequently asked questions about webhooks in .NET

Why should I verify the webhook signature over the raw request body? Because the signature is computed over the exact bytes the provider sent, and any re-serialization can change them. If you bind the request to a model and serialize it back, a new field, a different property order, or whitespace differences will make a valid payload fail verification. Read the raw bytes off the request stream, hash those, and compare with a constant-time function like CryptographicOperations.FixedTimeEquals so you do not leak how many bytes matched.

How do I stop processing the same webhook twice? Treat the provider's event id as a unique key in an inbox table and let the database reject duplicates. Insert the row and catch the unique-violation exception as a success rather than an error, because at-least-once delivery guarantees you will see repeats. That keeps the handler idempotent without a distributed lock or an extra round trip to check existence first, and it survives two deliveries arriving at the same moment.

Should the webhook endpoint do the real work inline? No. Persist the event, return 200 within milliseconds, and process it in a background worker that drains the inbox. If you run business logic inline, a slow dependency causes a timeout, the provider retries, and you end up processing the same event concurrently. Fast acknowledge plus out-of-band processing keeps you off the provider's retry timer and keeps your endpoint responsive under a burst.

Do I still need a reconciliation job if my webhooks work? Yes, and it should be loud. Pull the provider's event list, feed it through the same idempotent inbox, and count how many events it had to heal. Alert when that count is above zero, and monitor the job so a silent failure of the reconcile itself cannot hide drift. Reconciliation is a safety net, and a silent safety net is the thing that eventually bites you.

What is the difference between a webhook and a pull-based event log? A webhook is the provider pushing an event to your URL, with at-least-once delivery, no ordering guarantee, and no way to replay history after the retention window. A pull-based log is you reading an ordered, cursor-addressed stream on a connection you opened, which gives you ordering, bootstrap from the beginning, tombstones for deletes, and resumability after downtime. Logs suit data replication, webhooks suit one-off side effects.

Further reading

• That Friday Afternoon When Our Stripe Webhooks Stopped
• Idempotency Is Easy Until the Second Request Is Different: A .NET Field Guide
• Webhook Security: HMAC Signatures and Replay Attack Prevention

---

Conclusion: Treat "The Valley of Webhooks" as a design checklist, not a migration plan. As a consumer, make your receiver boring and your reconciliation loud. As a provider, give people a log to pull.

Sources: The Valley of Webhooks · Hacker News discussion.