Monitoring .NET Applications With OpenTelemetry and Grafana

A hospital monitor for your app

Picture a patient in a hospital bed. Next to the bed sits a small machine with a screen. It shows the heartbeat, the oxygen level, and the temperature, all updating live. A nurse can walk past, glance at the screen, and know in one second whether the patient is fine or in trouble. The patient cannot talk much, but the machine speaks for them.

Your .NET application is a lot like that patient. It is busy doing work, and it cannot stop to tell you how it feels. You need a monitor that watches it all the time and shows the vital signs on a screen. When something hurts, you want a clear number or a red line, not a guessing game at 2 in the morning.

That monitor is what we build today. OpenTelemetry is the set of sensors we attach to the app. Grafana is the screen the nurse looks at. In between sit a few storage tools that remember the readings. By the end of this guide you will know what each piece does and how they fit together.

The three vital signs: metrics, traces, and logs

Good monitoring watches three kinds of data. People call them the three signals or the three pillars of observability. Each one answers a different question.

Think back to the hospital. Metrics are like the heartbeat number on the screen, steady and always counting. Traces are like following one drop of blood through the whole body to see where it gets stuck. Logs are like the nurse's written notes: "Patient coughed at 3:14, gave water." You want all three. A number alone tells you something is wrong, but a trace and a log tell you why.

Meet the team of tools

This setup uses a small team of open-source tools. Each has one job. Knowing the job of each one makes the whole thing easy to remember.

The important idea is this: Grafana stores nothing by itself. It is just the glass screen. The real memory lives in Prometheus, Tempo, and Loki. Grafana asks them questions and paints the answers. This is why people often call the storage tools "the LGTM stack" (Loki, Grafana, Tempo, Mimir or Prometheus).

How the pieces connect

Before we write code, look at the full picture once. Your app speaks one language, called OTLP (the OpenTelemetry Protocol). It sends everything to the Collector. The Collector is the traffic police: it looks at each item and sends metrics to Prometheus, traces to Tempo, and logs to Loki. Grafana then reads from all three.

Why send to a Collector instead of straight to each storage tool? Three good reasons. First, your app stays simple: it knows only one address. Second, you can change backends later without touching code. If you switch from Tempo to another trace tool, you edit the Collector, not the app. Third, the Collector can clean and batch the data, which is kinder to your servers.

Step 1: Add the OpenTelemetry packages

We start in a normal ASP.NET Core project. On .NET 10, which is the current LTS release, the OpenTelemetry libraries work out of the box. Add these NuGet packages. They are all free and open source.

// Run these in your project folder:
// dotnet add package OpenTelemetry.Extensions.Hosting
// dotnet add package OpenTelemetry.Exporter.OpenTelemetryProtocol
// dotnet add package OpenTelemetry.Instrumentation.AspNetCore
// dotnet add package OpenTelemetry.Instrumentation.Http
// dotnet add package OpenTelemetry.Instrumentation.Runtime

The first two packages are the heart of it. The "Instrumentation" packages are helpers that watch common things for you. The ASP.NET Core one traces every incoming request. The Http one traces every outgoing call. The Runtime one reports numbers about memory and garbage collection. You do not have to write that code yourself.

Step 2: Turn on the three signals

Now open Program.cs. We tell the app to collect traces, metrics, and logs, and to send all of them over OTLP. The single UseOtlpExporter call wires up the exporter for every signal at once, which is the modern, simple way.

using OpenTelemetry.Logs;
using OpenTelemetry.Metrics;
using OpenTelemetry.Resources;
using OpenTelemetry.Trace;
 
var builder = WebApplication.CreateBuilder(args);
 
var serviceName = "shop-api";
 
builder.Services.AddOpenTelemetry()
    .ConfigureResource(r => r.AddService(serviceName))
    .WithTracing(tracing => tracing
        .AddAspNetCoreInstrumentation()
        .AddHttpClientInstrumentation())
    .WithMetrics(metrics => metrics
        .AddAspNetCoreInstrumentation()
        .AddHttpClientInstrumentation()
        .AddRuntimeInstrumentation())
    .UseOtlpExporter(); // sends traces, metrics, and logs
 
// Send logs through OpenTelemetry too
builder.Logging.AddOpenTelemetry(logging =>
{
    logging.IncludeScopes = true;
    logging.IncludeFormattedMessage = true;
});
 
var app = builder.Build();

A few words about what this does. AddService(serviceName) stamps every reading with your app's name, so in Grafana you can tell shop-api apart from payment-api. The WithTracing block turns on traces, WithMetrics turns on numbers, and the logging block sends your ILogger messages out as well. The UseOtlpExporter call is the one line that ships it all.

By default OTLP goes to http://localhost:4317 using gRPC. To point it somewhere else, you do not change code. You set an environment variable, which keeps the app clean.

// In appsettings or environment, no code change needed:
// OTEL_EXPORTER_OTLP_ENDPOINT = http://collector:4317

Step 3: Add your own custom metric and trace

The automatic instrumentation is great, but the best signals come from your own business. Maybe you want to count how many orders were placed, or measure how long a payment took. You create a Meter for numbers and an ActivitySource for traces. These are built into .NET, so no extra library is needed for the API itself.

using System.Diagnostics;
using System.Diagnostics.Metrics;
 
public class OrderService
{
    // A meter to publish custom numbers
    private static readonly Meter Meter = new("Shop.Orders");
    private static readonly Counter<long> OrdersPlaced =
        Meter.CreateCounter<long>("orders.placed");
 
    // A source to create custom traces
    private static readonly ActivitySource Activity = new("Shop.Orders");
 
    public async Task PlaceOrderAsync(string itemId)
    {
        using var span = Activity.StartActivity("PlaceOrder");
        span?.SetTag("item.id", itemId);
 
        await SaveToDatabaseAsync(itemId);
 
        OrdersPlaced.Add(1); // the number goes up by one
    }
 
    private Task SaveToDatabaseAsync(string itemId) => Task.CompletedTask;
}

For the SDK to notice these, you register their names. Add .AddMeter("Shop.Orders") to the metrics block and .AddSource("Shop.Orders") to the tracing block in Program.cs. Now every order shows up as a counter in Prometheus, and every PlaceOrder shows up as a span in Tempo.

Step 4: Run the storage tools with Docker

You do not install Prometheus, Tempo, Loki, and Grafana by hand. You run them with Docker Compose. Here is a simple shape of the file. It is YAML, not C#, but it is short and easy to read.

// docker-compose.yml (shown as text for clarity)
// services:
//   collector:  image otel/opentelemetry-collector  ports 4317, 4318
//   prometheus: image prom/prometheus               port 9090
//   tempo:      image grafana/tempo                 port 3200
//   loki:       image grafana/loki                  port 3100
//   grafana:    image grafana/grafana               port 3000

Once these are running, you open Grafana in a browser at http://localhost:3000. You add three data sources: Prometheus, Tempo, and Loki, each pointing to its container. From then on Grafana can draw any chart you ask for.

Step 5: Build your first dashboard

A dashboard is just a screen full of panels. Each panel asks one question and draws the answer. In Grafana you pick a data source, type a small query, choose a chart shape, and save. You can start with these four panels:

• Requests per second from Prometheus, drawn as a line. This is the heartbeat.
• Error rate from Prometheus, drawn as a line that should stay near zero.
• Slowest traces from Tempo, so you can click into any slow request.
• Recent error logs from Loki, filtered to show only failures.

The real magic is that the three connect. In a Tempo trace you can jump straight to the matching logs in Loki, because they share the same trace id. So you see a slow request, click it, and read the exact log lines from that moment. That is the dream the nurse never had: the heartbeat and the notes side by side.

Reading a trace like a story

A trace is made of spans. Each span is one unit of work with a start time and an end time. The top span is the whole request. Inside it are child spans for the database call, the cache check, the call to another service. Drawn out, a trace looks like a waterfall, and the long bars show you where the time went.

If the payment step took four seconds, the bar for "Charge card" would be long and obvious. You would not have to guess. You would see it. That is the whole point of tracing: it turns a vague "the site is slow" into a precise "the payment service is the slow part."

A simple checklist before you ship

Monitoring is only useful if it is set up cleanly. Here is a short list to run through before you call it done.

• Give every service a clear name with AddService, so you can tell them apart.
• Turn on all three signals, even if you only watch metrics at first. The data is cheap to collect.
• Send to a Collector, not straight to storage, so you can change tools later without code changes.
• Use sampling for traces in busy apps, so you keep a useful fraction instead of every single request.
• Never put secrets like passwords into span tags or logs. Anyone who can see Grafana can see them.
• Add a couple of alerts in Grafana, so the screen calls you instead of you having to stare at it.

A note on cost and overhead

A fair question: does all this slow the app down? The honest answer is a little, but you control how much. Metrics are almost free. Logs cost as much as you log, so log what matters and skip the noise. Traces cost the most, which is why sampling exists: under heavy traffic you might record one in ten requests and still spot every pattern. The data leaves your app in the background, in batches, so your users never wait on it. For nearly every team, the time saved during an outage is worth far more than the tiny cost of collection.

Quick recap

• OpenTelemetry is the sensor inside your .NET app. It collects three signals: metrics, traces, and logs.
• Grafana is the screen. It stores nothing by itself; it only reads and draws.
• Storage lives in Prometheus (metrics), Tempo (traces), and Loki (logs).
• Your app speaks OTLP to a Collector, which forwards each signal to the right store.
• Add the OpenTelemetry packages, call AddOpenTelemetry() with tracing and metrics, and finish with UseOtlpExporter().
• Create custom signals with a Meter for numbers and an ActivitySource for traces.
• Traces are made of spans; the long bars show you where time was spent.
• Use sampling and a few alerts, keep secrets out of logs, and you have a healthy, watchable app.

References and further reading

• .NET Observability with OpenTelemetry — Microsoft Learn
• OpenTelemetry .NET — official docs
• OpenTelemetry .NET — Exporters
• Grafana — sending OTLP data
• OTLP Exporter for OpenTelemetry .NET (GitHub)