The Idempotent Consumer Pattern in .NET (And Why You Need It)

Imagine you are buying a train ticket on your phone. You tap "Pay 250 rupees" and the app spins. The signal drops in a tunnel. You are not sure the booking went through, so when the signal returns you tap "Pay" again. A good booking system makes sure you get one ticket and one charge, even though your phone shouted the same request twice.

That calm, "I already did this, so I will quietly ignore the repeat" behaviour is the heart of the idempotent consumer pattern. In a .NET system built on messages, the same story plays out constantly. A message can arrive twice. If your handler is not careful, you double-charge a card, ship two parcels, or send two confirmation emails. This guide shows you why duplicates are unavoidable, and how to handle them gently and safely.

A quick word on what "idempotent" means

The word is long, but the idea is small.

An action is idempotent if doing it many times leaves you in the same place as doing it once.

Think of a ceiling fan switch set to "ON". Flip it to "ON" again and nothing changes; it is already on. Or think of saving your home address in an app. Save the same address twice and you still have one address. These are idempotent.

Now the opposite. "Add 250 rupees to this account." Do it twice and you added 500. "Send a thank-you SMS." Do it twice and the customer gets two messages. These are not idempotent, and these are exactly the actions that hurt when a message repeats.

Our whole job is to make a risky, non-idempotent action behave like the fan switch.

Why you need it: duplicates are a promise, not an accident

Here is the part many people miss. Duplicate messages are not a sign that something is broken. They are a direct result of a promise your broker makes on purpose.

Almost every message broker, RabbitMQ, Azure Service Bus, Amazon SQS, and Apache Kafka, offers at-least-once delivery. In plain words: "I will hand your message to a consumer at least once. Sometimes more than once. But never zero times."

Why would a broker ever send the same message twice? Because of acknowledgements. When a consumer finishes a message, it sends an ACK back to the broker that means "done, you can forget this one". But picture this: the consumer does the work, then crashes a split second before the ACK leaves. The broker waits, hears nothing, assumes the work never happened, and sends the message again to be safe.

So duplicates come from healthy, everyday events: a pod restart during a deploy, a brief network hiccup, a slow handler that trips a retry timer, a Kafka consumer rebalance. None of these are bugs. They are normal life in a distributed system.

This is the real answer to "why you need it". You are not protecting against a rare failure. You are protecting against the guaranteed behaviour of the tools you chose. Skipping idempotency is choosing to ship a known bug.

But can't I just ask for exactly-once?

People often wish for exactly-once delivery, where the broker swears no duplicate ever appears. The honest truth is that true exactly-once delivery across a network is impossible. There is always a tiny window where the consumer and broker can disagree about what happened.

The working answer the whole industry uses is this:

At-least-once delivery + an idempotent consumer = effectively-once processing.

The wire may carry the message twice. Your consumer simply refuses to act on it twice. From the outside, the result looks exactly once.

The core idea: remember what you already did

The simplest path to idempotency is to remember every message you have processed.

Each message carries a unique id, usually called a MessageId or an idempotency key. Before doing the real work, you ask one question: "Have I seen this id before?"

• If no: do the work, and store the id.
• If yes: skip it; it is a duplicate.

We keep these ids in a small database table, often named inbox or processed_messages.

It sounds easy, and the idea is. The danger is in the details: the check, the work, and the save must hold together as one unit. If a crash sneaks between them, you are back to double effects. We will close that gap with a transaction.

A naive first try, and its hidden bug

Let us write a handler that charges a customer when an OrderPlaced message arrives. This is the kind of code people write on day one.

public sealed class OrderPlacedHandler
{
    private readonly AppDbContext _db;
    private readonly IPaymentService _payments;
 
    public OrderPlacedHandler(AppDbContext db, IPaymentService payments)
    {
        _db = db;
        _payments = payments;
    }
 
    public async Task HandleAsync(OrderPlaced message, CancellationToken ct)
    {
        // Step 1: have we seen this message before?
        bool alreadyDone = await _db.ProcessedMessages
            .AnyAsync(p => p.MessageId == message.MessageId, ct);
 
        if (alreadyDone)
            return; // duplicate, skip it
 
        // Step 2: do the real work
        await _payments.ChargeAsync(message.CustomerId, message.Amount, ct);
 
        // Step 3: remember this message
        _db.ProcessedMessages.Add(new ProcessedMessage
        {
            MessageId = message.MessageId,
            ProcessedOnUtc = DateTime.UtcNow
        });
 
        await _db.SaveChangesAsync(ct);
    }
}

This reads as if it is correct, but look closely at the gap between Step 2 (charge the card) and Step 3 (save the id). If the process dies right there, the card is charged but the id is never stored. The broker redelivers, Step 1 finds nothing, and the card is charged again. The very table that was meant to protect you sat empty at the worst moment.

The fix is to make the business change and the id record live or die together. That is what a transaction is for.

The safe version: one transaction (the lazy approach)

We wrap the business change and the id save in a single database transaction. If anything fails, the whole thing rolls back and the database stays clean. Because we save the id only when the work succeeds, this is often called the lazy approach. It is the one most teams should reach for: it needs one less round trip and is easy to reason about.

public async Task HandleAsync(OrderPlaced message, CancellationToken ct)
{
    await using var tx = await _db.Database.BeginTransactionAsync(ct);
 
    bool alreadyDone = await _db.ProcessedMessages
        .AnyAsync(p => p.MessageId == message.MessageId, ct);
 
    if (alreadyDone)
    {
        await tx.CommitAsync(ct);
        return; // duplicate, nothing more to do
    }
 
    // The business change lives in the SAME database and SAME transaction
    _db.Orders.Add(new Order(message.OrderId, message.CustomerId, message.Amount));
 
    _db.ProcessedMessages.Add(new ProcessedMessage
    {
        MessageId = message.MessageId,
        ProcessedOnUtc = DateTime.UtcNow
    });
 
    await _db.SaveChangesAsync(ct);
    await tx.CommitAsync(ct);
}

Notice the key move. The saved Order and the id record now sit in the same database and the same transaction. They cannot disagree. Either both land, or neither does. The crash gap from the naive version is gone.

There is still one tiny race left: two copies of the same message could run at the exact same instant, both pass the AnyAsync check, and both try to insert the same id. We close that next.

Eager versus lazy: two honest ways to record the id

There are two timings for saving the message id, and it helps to know both.

With the eager approach you claim the id up front. The upside is that two racing copies cannot both start. The downside is real: if the handler throws after you saved the id, that id now says "done" even though nothing happened. The redelivery will see it and skip the work forever. So the eager path forces you to write careful cleanup that removes the id when the handler fails.

With the lazy approach the id is saved only on success, inside the same transaction as the work. A failure rolls everything back, including the id, so a later redelivery correctly tries again. There is no cleanup to remember. Unless you have a specific reason, prefer lazy.

Let the database be the referee

The cleanest way to kill the last race is to make MessageId the primary key of the inbox table. Now the database itself refuses to store the same id twice. You do not even need to check first. You just try to insert and let the database say "no" if it is a duplicate.

public async Task<bool> TryMarkProcessedAsync(Guid messageId, CancellationToken ct)
{
    _db.ProcessedMessages.Add(new ProcessedMessage
    {
        MessageId = messageId,          // primary key
        ProcessedOnUtc = DateTime.UtcNow
    });
 
    try
    {
        await _db.SaveChangesAsync(ct);
        return true; // we are the first to process it
    }
    catch (DbUpdateException ex) when (IsUniqueViolation(ex))
    {
        return false; // someone already claimed it, so it is a duplicate
    }
}

This is powerful because the database is a single, trusted referee. Even if two consumers race in the same millisecond, only one insert wins. The loser catches the unique-violation error, learns that it lost, and skips the work. No double charge, no extra coordination.

The table itself is tiny:

public sealed class ProcessedMessage
{
    public Guid MessageId { get; set; }     // primary key
    public DateTime ProcessedOnUtc { get; set; }
}

What about the same message going to many handlers?

In real systems, one event often fans out to several consumers. An OrderPlaced message might go to a billing handler, an email handler, and a warehouse handler. Each one needs its own idempotency check. If billing has processed the message, that does not mean email has.

So a single MessageId primary key is not enough here. You widen the key to a pair: the message id and the consumer name. The same message can be marked "done" once per consumer, but never twice for the same consumer.

public sealed class ProcessedMessage
{
    public Guid MessageId { get; set; }       // part of the key
    public string ConsumerName { get; set; } = default!; // part of the key
    public DateTime ProcessedOnUtc { get; set; }
}
 
// In OnModelCreating:
// modelBuilder.Entity<ProcessedMessage>()
//     .HasKey(p => new { p.MessageId, p.ConsumerName });

Now billing checks for the pair (messageId, "billing"), email checks (messageId, "email"), and they never step on each other.

When you do not need any of this

Not every consumer deserves an inbox table. Sometimes the action is already safe to repeat, and adding ceremony just wastes effort and database round trips. This is natural idempotency, and it is the nicest situation to be in.

Ask yourself a simple question about the action: if I run it twice, does the world look any different than running it once?

• "Set the order status to Shipped." Running it twice leaves the status at Shipped. Safe.
• "Refresh this cached value." Running it twice gives the same fresh value. Safe.
• "Update the search projection for order 42." Deterministic, so safe to repeat.

Compare those to "insert a payment row" or "send money", which leave a new mark every time. Those need a guard; the safe ones do not.

Spend your effort where it counts: on the handlers whose effects you cannot undo.

Choosing a good message id

Your entire defence rests on the id, so pick it carefully.

• It must be unique per logical message. A fresh Guid per published message works well.
• It must be stable across redeliveries. The redelivered copy must carry the same id as the original, or the check is useless.
• It should be set by the producer, once, when the message is first created. It must never be regenerated by the consumer.

Most brokers already attach a MessageId to every message for exactly this purpose. Use it, or set your own and carry it inside the message body.

Don't forget side effects outside the database

Your transaction protects your own database. It cannot un-send an email or un-call a payment gateway. So treat external effects with extra care.

The safest pattern is: do your local database work and commit first, then perform the external effect after the commit. And whenever the external service supports it, pass an idempotency key so the service itself dedupes for you. Stripe, for example, accepts an Idempotency-Key header and will not charge twice for the same key.

public async Task ConsumeAsync(OrderPlaced message, CancellationToken ct)
{
    await using var tx = await _db.Database.BeginTransactionAsync(ct);
 
    // The database is the referee: claim this id first
    if (!await TryMarkProcessedAsync(message.MessageId, ct))
    {
        await tx.CommitAsync(ct);
        return; // duplicate, safely ignored
    }
 
    // Local business change, same transaction as the id record
    _db.Orders.Add(new Order(message.OrderId, message.CustomerId, message.Amount));
    await _db.SaveChangesAsync(ct);
    await tx.CommitAsync(ct);
 
    // External effect AFTER commit, carrying its own idempotency key
    await _payments.ChargeAsync(
        message.CustomerId,
        message.Amount,
        idempotencyKey: message.MessageId.ToString(),
        ct);
}

Read it slowly. Claim the id, do the local work, commit both together, then reach out to the world last, with a key that lets the outside service protect itself too. Each layer guards the one before it.

Keep the inbox table small

The inbox table grows forever if you never trim it. The good news is you do not need to keep ids for all time. A duplicate almost always shows up within minutes or hours, not weeks. A simple background job can delete old rows.

public async Task CleanupOldEntriesAsync(CancellationToken ct)
{
    var cutoff = DateTime.UtcNow.AddDays(-7);
 
    await _db.ProcessedMessages
        .Where(p => p.ProcessedOnUtc < cutoff)
        .ExecuteDeleteAsync(ct);
}

Keep a window comfortably longer than your broker's maximum retry time. Seven days is a safe, common choice. Add an index on ProcessedOnUtc so the cleanup query stays fast.

What the .NET ecosystem gives you

When you wire this up in real projects, you usually reach for a messaging library. A few honest notes for 2026:

• MassTransit moved to a commercial license. It is excellent and ships a built-in inbox/outbox that does most of this for you, but check the pricing before you depend on it.
• MediatR also went commercial. It is an in-process mediator rather than a broker, but the same licensing caution applies.
• Wolverine is open source and offers a durable inbox that deduplicates messages for you, a strong free option.
• NServiceBus has long offered an outbox feature that delivers effectively-once processing and is a mature, paid choice.

You can also build everything by hand with plain EF Core and a broker client, which is exactly what the code in this guide does. For learning, hand-rolling it is the best way to truly understand the moving parts. Whatever you pick, the rule never changes: the broker gives at-least-once delivery, and your consumer turns that into effectively-once with a stored id and a single transaction.

Common mistakes to avoid

Even careful teams trip over these.

• Checking and saving in separate transactions. This reopens the crash gap. Keep them in one transaction.
• Acknowledging before the work is committed. If you ACK first and then crash, the message is gone and the work never happened. Commit, then ACK.
• Using a new id on each redelivery. If every copy gets a fresh id, every copy looks new and you process them all. Fix the id at the producer.
• Sharing one id key across many consumers. Fan-out needs the (MessageId, ConsumerName) pair, or one consumer's success wrongly silences another.
• Forgetting external effects. Your transaction cannot un-send an email or un-charge a card. Do those after commit and pass an idempotency key.

Quick recap

• Brokers offer at-least-once delivery, so duplicate messages are guaranteed, not rare. That is why you need this pattern.
• Idempotent means doing something many times has the same effect as doing it once.
• Give every message a unique, stable id set by the producer.
• Store processed ids in an inbox table, with the id as the primary key.
• Keep the id record and the business change in one transaction so they never disagree.
• Prefer the lazy approach (save the id on success) over the eager one (save first, clean up on failure).
• Let the database be the referee by catching unique-key violations to detect duplicates.
• For fan-out, key on (MessageId, ConsumerName) so each consumer dedupes on its own.
• Skip the table when the action is naturally idempotent, like "set status to Shipped".
• Do external side effects after commit, and give them their own idempotency key.
• True exactly-once delivery is impossible, but at-least-once plus an idempotent consumer gives effectively-once processing.

References and further reading

• Microservices.io - Idempotent Consumer Pattern
• Milan Jovanovic - The Idempotent Consumer Pattern in .NET (And Why You Need It)
• Milan Jovanovic - Implementing the Inbox Pattern for Reliable Message Consumption
• Confluent - Message Delivery Guarantees for Apache Kafka
• Wolverine - Durable Inbox and Outbox Messaging