Idempotent Consumer: Handling Duplicate Messages in .NET

Imagine you are paying for your morning tea using a UPI app. You tap "Pay 10 rupees". The screen freezes. You are not sure if the money left your account, so you tap "Pay" again. Now you are worried: did the shopkeeper get charged twice?

A good payment system makes sure that even if your phone sends the same request two times, the money moves only once. The second request is quietly ignored. That is the whole idea behind the Idempotent Consumer pattern.

In message-driven .NET systems, the same thing happens all the time. A message can arrive twice. If your code is not careful, you might charge a card twice, send two emails, or ship two parcels. This guide shows you how to stop that, in simple steps.

What "idempotent" really means

The word sounds scary, but the idea is gentle.

An operation is idempotent if doing it many times has the same effect as doing it once.

Here are everyday examples:

• Pressing a lift button five times. The lift still comes once.
• Switching a light switch to "ON" when it is already on. Nothing new happens.
• Setting your age to 12. Setting it to 12 again changes nothing.

Now compare that to a non-idempotent action:

• "Add 100 rupees to the balance." Do it twice and you added 200. Oops.
• "Send a thank-you email." Do it twice and the customer gets two emails. Annoying.

Our goal is to make message handlers behave like the lift button, not like the "add 100 rupees" action.

Why duplicate messages happen at all

You might ask: "If duplicates cause bugs, why does the broker send them?"

The answer is about a promise the broker makes. Most message brokers, like RabbitMQ, Azure Service Bus, Amazon SQS, and Apache Kafka, give you at-least-once delivery. This means: "I will deliver your message at least once. Maybe more, but never zero."

Why "maybe more"? Because of acknowledgements (ACKs).

When a consumer finishes a message, it sends an ACK to the broker. The ACK says "done, you can forget this one". But what if the consumer does the work, then crashes before sending the ACK? The broker waits, hears nothing, and thinks the work was never done. So it sends the message again to be safe.

So duplicates are not a rare bug. They are a normal part of healthy messaging systems. Network blips, consumer restarts, retries, and rebalances all cause them. We must plan for them.

Why not just use "exactly-once"?

People often wish for exactly-once delivery, where the broker guarantees no duplicates ever. Sadly, true exactly-once delivery across a network is impossible. There is always a tiny moment where the consumer and broker can disagree about what happened.

The honest, working answer used across the industry is:

At-least-once delivery + an idempotent consumer = effectively-once processing.

The broker may send duplicates. Your consumer just refuses to act on them twice. The visible result is "exactly once", even though the wires saw the message more than once.

The core idea: remember what you have done

The simplest way to be idempotent is to remember every message you have already processed.

Every message should carry a unique id, often called a MessageId or an idempotency key. Before you do the real work, you ask one question: "Have I seen this id before?"

• If no: process it, and save the id.
• If yes: skip it. It is a duplicate.

We store these ids in a database table. People usually call this table inbox or processed_messages.

This sounds easy, and it mostly is. But there is one trap. The check, the work, and the save must happen together as one unit. If they do not, a crash in the middle can still cause trouble. We will fix that with a database transaction soon.

A first, naive version (and its bug)

Let us write a simple handler. We will pretend we are charging a customer when an OrderPlaced message arrives.

public sealed class OrderPlacedHandler
{
    private readonly AppDbContext _db;
    private readonly IPaymentService _payments;
 
    public OrderPlacedHandler(AppDbContext db, IPaymentService payments)
    {
        _db = db;
        _payments = payments;
    }
 
    public async Task HandleAsync(OrderPlaced message, CancellationToken ct)
    {
        // Step 1: have we seen this message before?
        bool alreadyDone = await _db.ProcessedMessages
            .AnyAsync(p => p.MessageId == message.MessageId, ct);
 
        if (alreadyDone)
            return; // duplicate, skip it
 
        // Step 2: do the real work
        await _payments.ChargeAsync(message.CustomerId, message.Amount, ct);
 
        // Step 3: remember this message
        _db.ProcessedMessages.Add(new ProcessedMessage
        {
            MessageId = message.MessageId,
            ProcessedOnUtc = DateTime.UtcNow
        });
 
        await _db.SaveChangesAsync(ct);
    }
}

This looks correct, but it has a quiet bug. Look at the gap between Step 2 (charge the card) and Step 3 (save the id). If the app crashes right there, the customer is charged, but the id is never saved. When the broker redelivers the message, Step 1 finds nothing, and we charge again.

The fix is to make sure the charge and the id save either both happen or both fail. That is what transactions are for.

The safe version: one transaction

We wrap the "save the id" and the "business change" in a single database transaction. If anything fails, the whole thing rolls back, leaving the database clean.

public async Task HandleAsync(OrderPlaced message, CancellationToken ct)
{
    await using var tx = await _db.Database.BeginTransactionAsync(ct);
 
    bool alreadyDone = await _db.ProcessedMessages
        .AnyAsync(p => p.MessageId == message.MessageId, ct);
 
    if (alreadyDone)
    {
        await tx.CommitAsync(ct);
        return; // duplicate, nothing to do
    }
 
    // Business change lives in the same database, same transaction
    _db.Orders.Add(new Order(message.OrderId, message.CustomerId, message.Amount));
 
    _db.ProcessedMessages.Add(new ProcessedMessage
    {
        MessageId = message.MessageId,
        ProcessedOnUtc = DateTime.UtcNow
    });
 
    await _db.SaveChangesAsync(ct);
    await tx.CommitAsync(ct);
}

Notice something important. The business change (saving the Order) and the id record now live in the same database and the same transaction. They cannot disagree. Either both are saved, or neither is.

There is still a small race: two copies of the same message could run at the exact same time, both pass the AnyAsync check, and both try to insert the same id. We solve that next.

Let the database be the referee

The cleanest trick is to make the MessageId the primary key of the inbox table. The database will then refuse to store the same id twice. You do not even need to check first; you just try to insert and let the database say "no" if it is a duplicate.

public async Task<bool> TryMarkProcessedAsync(Guid messageId, CancellationToken ct)
{
    _db.ProcessedMessages.Add(new ProcessedMessage
    {
        MessageId = messageId,          // primary key
        ProcessedOnUtc = DateTime.UtcNow
    });
 
    try
    {
        await _db.SaveChangesAsync(ct);
        return true; // we are the first to process it
    }
    catch (DbUpdateException ex) when (IsUniqueViolation(ex))
    {
        return false; // someone already processed it, it is a duplicate
    }
}

This is powerful because the database is a single, trusted referee. Even if two consumers race at the same millisecond, only one insert wins. The loser catches the unique-violation error and knows it lost, so it skips the work. No double charge.

The table itself is tiny:

public sealed class ProcessedMessage
{
    public Guid MessageId { get; set; }     // primary key
    public DateTime ProcessedOnUtc { get; set; }
}

Three common strategies compared

There is more than one way to be idempotent. Picking the right one depends on your situation.

Natural idempotency is the nicest when you can get it. For example, an operation like "set the order status to Paid" is already safe to repeat, because the second time changes nothing. But an operation like "add a payment row" is not naturally safe, so it needs the inbox table to guard it.

Here is a quick way to decide:

How this relates to the Inbox pattern

You may have heard of the Inbox pattern. It is a close cousin of the idempotent consumer, and the two often get mixed up. Here is the simple difference.

• The idempotent consumer is the idea: process each message only once.
• The Inbox pattern is one popular way to do it: write incoming messages into a database table first, then process them from there in the background.

The Inbox pattern treats your database as the reliable log of what came in. The consumer's only job at first is to save the message safely. A separate worker then reads the inbox and does the real work, marking each row as done. Because the save and the processing both live in your database, you keep that all-important single-transaction safety.

If you want a deeper walk-through of that approach, see the related guide on implementing the inbox pattern.

A note on the .NET messaging ecosystem

When wiring this up in real projects, you will reach for a messaging library. A few honest notes for 2026:

• MassTransit and MediatR are now commercially licensed. They are still excellent, but check the license and pricing before you depend on them in production. MassTransit even ships built-in inbox/outbox support that does much of this for you.
• NServiceBus has long offered an "outbox" feature that gives effectively-once processing, and it is a mature option.
• You can also build this by hand with plain EF Core and a broker client, which is exactly what the code above does. For learning, hand-rolling it is the best way to understand the moving parts.

Whatever you choose, the underlying rule does not change: the broker gives at-least-once delivery, and your consumer must turn that into effectively-once with a stored id and a single transaction.

Choosing a good message id

Your whole defence rests on the id, so pick it well.

• It must be unique per logical message. A new Guid per published message works great.
• It must be stable across redeliveries. The redelivered copy must carry the same id as the first copy, or the check is useless.
• It should come from the producer, set once when the message is first created, not generated again by the consumer.

Most brokers give every message a MessageId property for exactly this purpose. Use it, or set your own and carry it in the message body.

Don't forget to clean up

The inbox table grows forever if you never trim it. You do not need to keep ids for all time. A duplicate almost always arrives within minutes or hours, not weeks. So a simple background job can delete old rows.

public async Task CleanupOldEntriesAsync(CancellationToken ct)
{
    var cutoff = DateTime.UtcNow.AddDays(-7);
 
    await _db.ProcessedMessages
        .Where(p => p.ProcessedOnUtc < cutoff)
        .ExecuteDeleteAsync(ct);
}

Keep a window that is comfortably longer than your broker's maximum retry time. Seven days is a safe, common choice. Add an index on ProcessedOnUtc so the cleanup stays fast.

Common mistakes to avoid

Even careful teams trip over these. Watch out for them.

• Checking and saving in separate transactions. This brings back the crash gap. Keep them together.
• Acknowledging before the work is committed. If you ACK first and then crash, the message is gone and the work never happened. Commit, then ACK.
• Using a new id on redelivery. If each delivery gets a fresh id, every copy looks "new" and you process them all. The id must be fixed at the producer.
• Forgetting side effects outside the database. Sending an email or calling another service is not covered by your database transaction. For those, prefer natural idempotency or a provider that supports an idempotency key.

That last point deserves a moment. Your transaction protects your own database. It cannot un-send an email. So for external effects, send them after the commit, and design them to tolerate a repeat, or pass an idempotency key to the external service so it dedupes for you.

Putting it all together

Here is the shape of a complete, safe handler that ties the ideas together.

public async Task ConsumeAsync(OrderPlaced message, CancellationToken ct)
{
    await using var tx = await _db.Database.BeginTransactionAsync(ct);
 
    // Database is the referee: try to claim this id first
    if (!await TryMarkProcessedAsync(message.MessageId, ct))
    {
        await tx.CommitAsync(ct);
        return; // duplicate, safely ignored
    }
 
    // Safe business change, same transaction as the id record
    _db.Orders.Add(new Order(message.OrderId, message.CustomerId, message.Amount));
    await _db.SaveChangesAsync(ct);
 
    await tx.CommitAsync(ct);
 
    // External effects go AFTER commit, and carry their own key
    await _email.SendOrderConfirmationAsync(message.OrderId, ct);
}

Read it slowly. Claim the id, do the local work, commit both together, then do the outside-world work last. Each layer protects the one before it.

Quick recap

• Brokers use at-least-once delivery, so duplicate messages are normal, not a bug.
• Idempotent means doing something many times has the same effect as doing it once.
• Give every message a unique, stable id set by the producer.
• Store processed ids in an inbox (or processed_messages) table, with the id as the primary key.
• Keep the id record and the business change in one transaction, so they never disagree.
• Let the database be the referee by catching unique-key violations to detect duplicates.
• Commit first, then ACK. Do external side effects after commit, and give them their own idempotency key.
• Clean up old inbox rows with a background job to keep the table small.
• True exactly-once delivery is impossible, but at-least-once plus an idempotent consumer gives effectively-once processing.

References and further reading

• Microservices.io - Idempotent Consumer Pattern
• Microservices.io - Handling duplicate messages using the Idempotent consumer pattern
• Milan Jovanovic - Idempotent Consumer: Handling Duplicate Messages
• Milan Jovanovic - The Idempotent Consumer Pattern in .NET (And Why You Need It)
• Confluent - Message Delivery Guarantees for Apache Kafka