Load Testing Kafka Pipelines With C# and NBomber in .NET 10

The post office and the sorting room

Imagine a busy post office in your town. At the front counter, people drop off letters all day. Behind a wall, in a big sorting room, workers pick up those letters and send them to the right place.

Now picture the day before a festival. Suddenly everyone wants to send letters. The front counter takes them fast and drops them all into one big bin. But the sorting room only has two workers. The bin fills up. Letters pile higher and higher. The counter is fine, but the sorting room is drowning.

A Kafka pipeline works just like this. The producer is the front counter that drops messages into a topic (the big bin). The consumer is the sorting room that picks messages back out and does the real work. Most of the time, both keep up. But on a busy day, one side may fall behind.

Load testing is how we pretend the festival rush is here, before it actually arrives. We use C# and a tool called NBomber to push lots of messages very fast, then watch carefully to see which side slows down. This guide shows you how, step by step, in simple words.

What you will build

By the end, you will have a small console project that does three things:

1. Sends many messages into a Kafka topic as fast as it can.
2. Measures how many messages per second the producer handles.
3. Watches the consumer to see if it keeps up or falls behind.

You do not need to be an expert. If you can write a simple C# for loop, you can follow along. We are using .NET 10, which is the current Long Term Support release, and C# 14.

Step 1: Start Kafka on your machine

You need a Kafka broker to talk to. The easiest way is Docker. Save this as docker-compose.yml and run docker compose up -d.

services:
  kafka:
    image: apache/kafka:3.9.0
    ports:
      - "9092:9092"
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@localhost:9093
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

This modern Kafka image uses KRaft mode, which means it does not need ZooKeeper any more. One small container, and you are ready.

Step 2: Create the project and add packages

Open a terminal and run these commands. They make a new console app and add the two libraries we need.

dotnet new console -n KafkaLoadTest
cd KafkaLoadTest
dotnet add package NBomber
dotnet add package Confluent.Kafka

Confluent.Kafka is the official .NET client for Kafka. It is a thin, fast wrapper over the native librdkafka library, so it can push messages very quickly. NBomber is the load testing engine. It will call our producing code over and over and measure the timings.

Here is how the pieces fit together.

Step 3: Write a tiny producer

Before we load test, let us make sure we can send one message. A Kafka producer needs to know the broker address. Then it sends a message that has a key and a value.

using Confluent.Kafka;
 
var config = new ProducerConfig
{
    BootstrapServers = "localhost:9092",
    // Batch messages for higher throughput. The producer waits up
    // to LingerMs to gather more messages before sending a batch.
    LingerMs = 5,
    BatchSize = 100_000,
    Acks = Acks.Leader
};
 
using var producer = new ProducerBuilder<string, string>(config).Build();
 
var message = new Message<string, string>
{
    Key = "order-1",
    Value = """{ "orderId": 1, "amount": 250 }"""
};
 
var result = await producer.ProduceAsync("orders", message);
Console.WriteLine($"Sent to partition {result.Partition} at offset {result.Offset}");

Notice the LingerMs and BatchSize settings. These tell the producer to wait a tiny moment and group many messages into one network call. This is the single biggest trick for high throughput. It is like the post office waiting until a sack is full before driving it to the next town, instead of driving once per letter.

Step 4: Turn the producer into an NBomber scenario

Now the fun part. NBomber works around a simple idea: you write a scenario. A scenario describes what one virtual user does. NBomber then runs that scenario many times at the same time and measures every run.

Each run is wrapped in a Step. NBomber times the step and records if it passed or failed. Here we send one Kafka message per step.

using Confluent.Kafka;
using NBomber.CSharp;
 
var config = new ProducerConfig
{
    BootstrapServers = "localhost:9092",
    LingerMs = 5,
    BatchSize = 100_000,
    Acks = Acks.Leader
};
 
using var producer = new ProducerBuilder<string, string>(config).Build();
 
var scenario = Scenario.Create("produce_orders", async context =>
{
    var message = new Message<string, string>
    {
        Key = $"order-{context.ScenarioInfo.ThreadId}",
        Value = $$"""{ "orderId": {{context.InvocationNumber}}, "amount": 250 }"""
    };
 
    var result = await producer.ProduceAsync("orders", message);
    return Response.Ok(sizeBytes: result.Value.Length);
})
.WithoutWarmUp()
.WithLoadSimulations(
    // Ramp up to 200 virtual users over 30 seconds, then hold.
    Simulation.RampingInject(rate: 200, interval: TimeSpan.FromSeconds(1), during: TimeSpan.FromSeconds(30)),
    Simulation.Inject(rate: 200, interval: TimeSpan.FromSeconds(1), during: TimeSpan.FromSeconds(60))
);
 
NBomberRunner
    .RegisterScenarios(scenario)
    .Run();

Let us read this slowly:

• Scenario.Create gives the scenario a name and the action to repeat.
• Response.Ok(...) tells NBomber the step succeeded. If you return Response.Fail(), NBomber counts it as an error.
• WithLoadSimulations is the load shape. RampingInject slowly grows the rate so you do not slam the system from cold. Inject then holds a steady rate.

Two ways to shape load

NBomber gives you two families of load simulation, and it helps to know the difference.

For a Kafka producer, the open model (Inject) is usually what you want. You care about messages per second, not how many users are logged in. The post office cares how many letters arrive each minute, not how many people are standing in the building.

Step 5: Watch the consumer side

Sending fast is only half the story. Remember the sorting room. We must check that the consumer keeps up. A consumer reads messages and tracks an offset, which is its place in the topic. The gap between the newest message and the consumer's offset is the consumer lag.

Here is a small consumer that reads in a loop. Run it in a second terminal while the load test runs.

using Confluent.Kafka;
 
var config = new ConsumerConfig
{
    BootstrapServers = "localhost:9092",
    GroupId = "orders-worker",
    AutoOffsetReset = AutoOffsetReset.Earliest,
    EnableAutoCommit = true
};
 
using var consumer = new ConsumerBuilder<string, string>(config).Build();
consumer.Subscribe("orders");
 
var count = 0;
while (true)
{
    var result = consumer.Consume(TimeSpan.FromSeconds(1));
    if (result is null) continue;
 
    // Pretend the sorting work takes a little time.
    await Task.Delay(2);
    count++;
 
    if (count % 1000 == 0)
        Console.WriteLine($"Processed {count} messages");
}

That await Task.Delay(2) is on purpose. It pretends the consumer does real work, like saving to a database. If you raise this delay, the consumer slows down and lag grows. This is how you discover the breaking point: the moment the producer's speed beats the consumer's speed.

You can watch lag from the command line while the test runs:

docker exec -it kafkaloadtest-kafka-1 \
  /opt/kafka/bin/kafka-consumer-groups.sh \
  --bootstrap-server localhost:9092 \
  --describe --group orders-worker

The LAG column is the number to watch. If it keeps climbing, your sorting room is too small.

Step 6: Read the NBomber report

When the test finishes, NBomber prints a table and also writes an HTML report to a folder. The numbers you care about most are:

• RPS (requests per second): here it means messages per second the producer pushed.
• Latency percentiles like p50, p95, and p99. The p99 is the slowest 1 percent. If p99 is huge but p50 is small, most sends are fast but a few are very slow. That is a warning sign.
• Data transfer: total bytes moved.
• Fail count: any failed sends. This should be zero in a healthy run.

A common mistake is to celebrate a high producer RPS while ignoring lag. Pushing 50,000 messages per second is useless if the consumer reads only 5,000 per second. The bin overflows. Always read both numbers together.

Step 7: Find the breaking point

The real goal of load testing is not a single run. It is to find where things break. Do this by running the test a few times, raising the rate each time, and writing down what happens.

When lag starts climbing and will not come back down, you have found the limit of your current setup. Now you have choices:

• Add more consumers in the same group. Kafka shares the topic's partitions across them, so more workers read in parallel. This is hiring more sorting room staff.
• Add more partitions to the topic. A topic with one partition can have only one consumer doing real work. More partitions means more parallel reading.
• Make each message smaller or the work faster, so each worker handles more per second.

A note on libraries and licences

If you have read about .NET messaging before, you may have seen MassTransit and MediatR. Both are well known, but as of recent versions they have moved to a commercial licence for many uses. That is fine, but it is good to know before you build on them.

The tools in this guide are friendlier for learning. The Confluent.Kafka client is free and open source. NBomber has a free open source core that covers single-machine testing like ours; its paid tier adds cluster runs and extra reporting. For a laptop test, you pay nothing.

Common mistakes to avoid

• Testing producer only. Always run a consumer too, and watch lag. The producer alone gives a falsely happy picture.
• No warm-up thought. The first few sends are slow because connections are still opening. We used WithoutWarmUp() here for clarity, but for cleaner numbers you can add a short warm-up so cold-start spikes do not pollute your report.
• One partition. With one partition you can never test parallel consumers. Create the topic with several partitions from the start.
• Acks set to All on a single broker. On a one-broker test box, Acks.All and Acks.Leader behave similarly, but on a real cluster Acks.All waits for replicas and is slower. Test with the same setting you will use in production.
• Ignoring memory. A fast producer with a slow consumer fills broker disk and client buffers. Keep an eye on both.

Quick recap

• A Kafka pipeline has a producer (front counter) and a consumer (sorting room). Load testing pretends the festival rush has arrived.
• Use Confluent.Kafka to send and read messages, and NBomber to run your sending code fast and measure it.
• Set LingerMs and BatchSize on the producer for high throughput. Batching is the biggest speed trick.
• In NBomber, a scenario describes one action; WithLoadSimulations shapes the load. Use Inject for throughput tests.
• Always watch consumer lag. High producer RPS means nothing if the consumer cannot keep up.
• Read p95 and p99 latency, not just averages, and aim for a zero fail count.
• To raise capacity, add consumers, add partitions, or make each message cheaper to process.
• Confluent.Kafka and the NBomber core are free; MassTransit and MediatR are now commercially licensed for many uses.

References and further reading

• NBomber official documentation — how scenarios, steps, and load simulations work.
• NBomber on GitHub — source code and ready-to-run samples.
• Confluent .NET client for Apache Kafka — the official Confluent.Kafka library and examples.
• Kafka performance, latency, and throughput — Confluent — how to think about throughput and tuning.
• Optimize a Kafka producer for throughput — Confluent — batching, linger, and compression settings.