Introduction to Distributed Tracing With OpenTelemetry in .NET

Following a train across the country

Imagine you are taking a long train journey across India. You start in Mumbai, change trains at Nagpur, change again at Bhopal, and finally reach Delhi. Each station stamps your ticket with the time you arrived and the time you left. At the end of the trip, you can look at all the stamps together and see the full story: where you waited the longest, which leg of the trip was fast, and which one was slow.

Now think about what happens when you click a button in an app. That one click may travel through your web API, then to a database, then to a payment service, then to an email service, and finally back to you. You never see these stops. But if the click feels slow, you really want a list of stamps, just like the train ticket, that tells you which stop took too long.

Distributed tracing gives you exactly that. It stamps every stop in the journey of a request. When you put all the stamps together, you get a clear timeline. OpenTelemetry is the free, shared toolkit that does this stamping in .NET and many other languages. In this guide you will learn what traces and spans are, how they travel across services, and how to turn them on in a .NET app step by step.

We are using .NET 10, which is the current long-term support (LTS) release, but the same steps work on .NET 8 and .NET 9 too.

What is a trace, and what is a span?

Two words sit at the heart of tracing. Let us make them very clear.

• A trace is the whole journey of one request, from the moment it starts to the moment it finishes.
• A span is one single step inside that journey. A database call is a span. An HTTP call to another service is a span.

A trace is made of many spans joined together. The very first span is the root span. Every other span knows who its parent is. So the spans form a tree, like a family tree. Tools can draw this tree as a waterfall, where each bar shows how long a span took.

Here is the shape of a single trace.

Every span carries a few important pieces of data. The table below lists the ones you will see most often.

The trace id is the magic glue. Because every span in the journey shares the same trace id, a tool can gather them all and rebuild the full timeline, even when the spans came from different services on different machines.

How a request becomes a trace

Let us walk through what happens, step by step, when a request arrives.

In .NET you almost never create the root span by hand. The ASP.NET Core instrumentation does it for you. Your job is mostly to turn it on and, when you want, to add a few spans of your own for important steps.

The Activity API: tracing is already inside .NET

Here is a happy surprise. .NET already has a tracing engine built in. It lives in the System.Diagnostics namespace. In .NET, a span is called an Activity, and the thing that creates activities is called an ActivitySource.

So the words map like this.

This matters because it means OpenTelemetry in .NET is not a heavy outside framework that fights with the runtime. It is a thin layer that listens to the activities .NET already produces and ships them out in the standard OTLP format. OTLP, the OpenTelemetry Protocol, is the shared wire format that almost every tracing tool understands.

Setting it up step by step

Now let us add tracing to a real app. We will keep it simple and clear.

Step 1: Add the packages

Add the core OpenTelemetry packages plus a couple of instrumentation packages. Instrumentation packages are small helpers that automatically create spans for common work, like incoming web requests and outgoing HTTP calls.

dotnet add package OpenTelemetry.Extensions.Hosting
dotnet add package OpenTelemetry.Exporter.OpenTelemetryProtocol
dotnet add package OpenTelemetry.Instrumentation.AspNetCore
dotnet add package OpenTelemetry.Instrumentation.Http

It is a good habit to use the latest stable versions, because the OpenTelemetry .NET libraries improve often.

Step 2: Turn on tracing in Program.cs

Now wire it up. The code below registers tracing, switches on the two instrumentation helpers, and sends the spans out using the OTLP exporter.

using OpenTelemetry.Resources;
using OpenTelemetry.Trace;
 
var builder = WebApplication.CreateBuilder(args);
 
builder.Services.AddOpenTelemetry()
    .ConfigureResource(resource => resource
        .AddService(serviceName: "orders-api"))
    .WithTracing(tracing => tracing
        .AddAspNetCoreInstrumentation()   // spans for incoming requests
        .AddHttpClientInstrumentation()    // spans for outgoing HTTP calls
        .AddSource("Orders.Custom")        // our own spans (see Step 3)
        .AddOtlpExporter());               // ship spans out via OTLP
 
var app = builder.Build();
 
app.MapGet("/orders", () => "Your orders are here");
 
app.Run();

A few things to notice. The AddService call gives your app a name, so your spans are easy to find in the tool. The two instrumentation lines mean you get spans for free, with no extra code. The AddOtlpExporter line sends everything to a collector or a tool. By default it sends to http://localhost:4317 using gRPC, which is where many local tools listen.

Step 3: Add your own span for a special step

The automatic spans are great, but sometimes you want a span around a piece of your own logic, like a price calculation. You create your own ActivitySource once and reuse it. The name must match the AddSource line from Step 2, or your spans will be ignored.

using System.Diagnostics;
 
public static class Tracing
{
    // The name here must match AddSource("Orders.Custom").
    public static readonly ActivitySource Source = new("Orders.Custom");
}
 
public class PriceCalculator
{
    public decimal Calculate(int quantity, decimal unitPrice)
    {
        // StartActivity opens a span. The "using" closes it at the end.
        using Activity? span = Tracing.Source.StartActivity("Calculate price");
 
        var total = quantity * unitPrice;
 
        // Tags become attributes you can see in the trace.
        span?.SetTag("order.quantity", quantity);
        span?.SetTag("order.total", total);
 
        return total;
    }
}

The using keyword is doing quiet but important work. When the method ends, the span is closed and its end time is recorded. The ? marks mean the code is safe even if no listener is collecting traces, in which case StartActivity returns null and nothing breaks.

How spans connect across services

This is the part that makes tracing distributed. When one service calls another, you want both services to put their spans into the same trace, not two separate ones. This is done with context propagation.

When service A calls service B over HTTP, OpenTelemetry adds a header named traceparent to the request. That header carries the trace id and the parent span id. Service B reads the header and continues the same trace.

The best news is that in .NET you do not write any of this by hand. The HttpClient instrumentation adds the traceparent header on the way out, and the ASP.NET Core instrumentation reads it on the way in. Both services just need OpenTelemetry turned on. They link up by themselves.

The traceparent header follows the W3C Trace Context standard. It looks like this: 00-7b2c...e91-a14f...02d-01. The middle two parts are the trace id and the parent span id. Because it is a shared standard, services written in different languages can join the same trace.

Sampling: you do not have to keep every trace

A busy app can create millions of traces every day. Storing all of them costs money and is rarely needed. Sampling is the rule that decides which traces to keep.

The simplest rule is "keep everything", which is perfect while you are learning. In production, teams often keep a slice, such as 10 percent, or keep every trace that had an error. The table below compares the common choices.

Setting a sampler is one line.

builder.Services.AddOpenTelemetry()
    .WithTracing(tracing => tracing
        .SetSampler(new TraceIdRatioBasedSampler(0.1))  // keep 10%
        .AddAspNetCoreInstrumentation()
        .AddOtlpExporter());

ParentBased is worth a special mention. It means a downstream service respects the decision made by the first service. If the first service decided to keep the trace, every later service keeps its part too. This stops you from ending up with half a trace.

The full picture across a system

Let us zoom out and see how all the pieces sit together in a small system with two services, a database, and a viewing tool.

Both the Orders API and the Payments API send their spans to the same place. Because the spans share a trace id, the viewer can stitch them into one waterfall. You then see, in one screen, that the request spent most of its time waiting on the bank gateway, for example. That insight is the whole point.

Here is the journey of the telemetry data itself, from your code to your eyes.

A quick word on how this fits the wider .NET world

OpenTelemetry is the modern, recommended path for tracing in .NET. It is open source and community owned, so there is no surprise license fee. This is a fair thing to flag, because some other popular .NET libraries, such as MediatR and MassTransit, have recently moved to commercial licensing for newer versions. OpenTelemetry has not, and the Activity API it builds on ships inside .NET itself. So you can lean on it without worrying about being charged later.

If you use .NET Aspire, you get a lot of this wiring for free. Aspire sets up OpenTelemetry tracing, metrics, and logging in its shared service defaults, and it ships with a dashboard that draws trace waterfalls out of the box. That makes Aspire a lovely way to see distributed traces on your own machine while you learn.

Reading a trace once it works

When you open a trace in a viewer, you will see a waterfall chart. Each row is a span. The longer the bar, the longer that step took. Here is how to read it like a detective.

• Look for the widest bar. That step took the most time. Start there.
• Check the gaps between bars. A gap can mean your code was waiting on something, like a lock or a slow disk.
• Open a span and read its attributes. The URL, the status code, and the row count often explain the slowness.
• Follow a span into the next service. Because the trace is shared, you can chase a slow call across service boundaries.

With a little practice, a slow request stops being a mystery. The trace points straight at the stop that needs your attention, exactly like the latest stamp on a train ticket.

Common mistakes to avoid

A few small traps catch most beginners. Keep these in mind.

• Forgetting AddSource. If you create your own ActivitySource but forget to register its name with AddSource, your custom spans simply will not appear. The name must match exactly.
• Wrong OTLP port. gRPC usually uses port 4317 and HTTP/protobuf usually uses 4318. If you point at the wrong one, nothing arrives. Check which one your tool listens on.
• Keeping everything in production. AlwaysOn is great for learning but can be expensive at scale. Switch to a ratio sampler when traffic grows.
• Putting secrets in attributes. Tags are visible to anyone who reads the trace. Never tag a password, a token, or a full credit card number.

Quick recap

• Distributed tracing follows one request through every stop it makes, like station stamps on a train ticket.
• A trace is the whole journey. A span (called an Activity in .NET) is one step inside it. Spans form a tree linked by a shared trace id.
• .NET already has a tracing engine in System.Diagnostics. OpenTelemetry is a thin layer over it that exports spans in the standard OTLP format.
• You turn it on with AddOpenTelemetry().WithTracing(...), add instrumentation for ASP.NET Core and HttpClient, and ship spans with AddOtlpExporter.
• Create your own spans with an ActivitySource, and register its name with AddSource so they are collected.
• Context propagation uses the traceparent header to join spans across services into one trace, and .NET does this for you automatically.
• Sampling decides which traces to keep. Use AlwaysOn while learning and a ratio or parent-based sampler in production.
• Read a trace as a waterfall: the widest bar is usually where the time went.

References and further reading

• OpenTelemetry .NET — official docs
• .NET Observability with OpenTelemetry — Microsoft Learn
• Distributed tracing concepts — Microsoft Learn
• OTLP Exporter for OpenTelemetry .NET — GitHub
• W3C Trace Context standard