Configuration and logging
Where an ASP.NET Core app gets its settings, where to keep secrets and how to validate configuration at startup, how to write structured logs and tie them to traces and metrics. Interviewers use this topic to check not only the API but also operational experience: what you do when “nothing is visible” in production.
Before this topic: Dependency injection.
How does configuration work: sources, their order and Section:Key keys?
Section titled “How does configuration work: sources, their order and Section:Key keys?”What it is
Configuration is a dictionary of string key-value pairs. The app reads it through IConfiguration, and providers fill it, one per source.
Default sources
WebApplication.CreateBuilder adds them in this order, and each next one overrides the previous ones:
appsettings.json— shared settings.appsettings.{Environment}.json— per-environment differences.- User Secrets — Development only.
- Environment variables.
- Command-line arguments.
How to use
- JSON nesting becomes a colon-separated key:
config["Payments:TimeoutSeconds"]. - In an environment variable, the colon is replaced with
__:Payments__TimeoutSeconds=60. GetValue<int>(...)converts the string to a type, andGetConnectionString("Default")reads theConnectionStrings:Defaultkey.
Common mistake
A typo in a key: config["Payment:Timeout"] returns null without an exception.
Takeaway: settings are a flat dictionary built from several sources, and the last source wins.
How it works
builder.Configurationis aConfigurationManager(.NET 6+): both a builder and the finished configuration, so you can read it right inProgram.cs.- Each provider loads its own dictionary. A key lookup goes from the last provider to the first until it finds a match.
- Keys are case-insensitive. Array elements are keys with indexes:
Payments:Currencies:0. - The command line accepts
--Key=value,--Key value,/Key valueandKey=value.
When to choose what
| What you configure | Where to keep it |
|---|---|
| Non-secret defaults | appsettings.json in the repository |
| Per-environment differences | appsettings.{Environment}.json or environment variables |
| Deployment-specific values, secrets | environment variables, a secret store |
| One-off runs, debugging | command-line arguments |
Pitfalls
- Your own source after
CreateBuilder.builder.Configuration.AddJsonFile(...)goes to the end of the list and overrides environment variables and the command line. - Arrays are merged by index. If the base file has three elements and the environment file has two, the third one stays from the base file.
Rule of thumb
After your own source, call AddEnvironmentVariables() and AddCommandLine(args) again so the deployment can still override everything.
Takeaway: the last provider wins, so the source order is part of the deployment contract.
Under the hood
ConfigurationManagerbuilds a provider as soon as a source is added. That is why values are available beforeBuild().- A provider is a case-insensitive dictionary of strings. Types appear only during binding:
ConfigurationBinderparses strings viaTypeConverterand reflection. - File providers with
reloadOnChange: truewatch the file, reread it and signal throughIChangeToken. The default template enables this forappsettings*.json.
Cost
- A key lookup walks the providers from the end. In hot code, values are bound to an object once through Options instead of reading
config["..."]on every request. - In containers and on network drives, file system events are unreliable. There you set
DOTNET_USE_POLLING_FILE_WATCHER=true: polling every four seconds.
Trade-offs
- In .NET 10 a JSON
nullbinds asnull, and an empty array[]binds as an empty array. Previouslynullbecame an empty string, so behavior changes when you migrate. - Reflection-based binding produces trimming warnings and is not compatible with Native AOT. The binding source generator (.NET 8+) is enabled with the
EnableConfigurationBindingGeneratorproperty.
How to check in production
((IConfigurationRoot)config).GetDebugView() shows each key’s value and the provider it came from. The overload with processValue masks secrets; without it, never expose the output.
Takeaway: configuration is a stack of string dictionaries, and the provider order together with the moment of binding determines what your code sees.
{ "Payments": { "TimeoutSeconds": 30, "Currencies": [ "UAH", "EUR" ] }, "ConnectionStrings": { "Default": "Host=localhost;Database=shop" }}var builder = WebApplication.CreateBuilder(args);
// Values are already merged from all sources, so you can read them before Build()int timeout = builder.Configuration.GetValue<int>("Payments:TimeoutSeconds");
var app = builder.Build();
app.MapGet("/config", (IConfiguration config) => new{ Timeout = timeout, // 30, or 60 with Payments__TimeoutSeconds=60 FirstCurrency = config["Payments:Currencies:0"], // "UAH" HasDb = config.GetConnectionString("Default") is not null, // key ConnectionStrings:Default Typo = config["Payment:Timeout"] // null: a typo, no exception});
app.Run();Running dotnet run --Payments:TimeoutSeconds=90 gives 90 even when the environment variable is set: the command line is added last.
- Which wins:
appsettings.Production.jsonor an environment variable? — The environment variable: its provider is added later. - Why do environment variables use
__instead of:? — Bash and some platforms don’t accept:in names;__is turned into:when read. - What happens if you add your own JSON file after
CreateBuilder? — It ends up last and overrides even the command line. - How do you override the second array element from the environment? — With an indexed key:
Payments__Currencies__1=USD. - How do you find out where a value came from? —
GetDebugView()onIConfigurationRootshows the provider of every key.
How do environments work, and where should you keep secrets?
Section titled “How do environments work, and where should you keep secrets?”What it is
The environment is a string the app reads at startup. The standard values are Development, Staging and Production; if nothing is set, it is Production.
How to use
- Locally the environment comes from
launchSettings.json, on a server from theASPNETCORE_ENVIRONMENTvariable. - In code you check
builder.Environment.IsDevelopment()orapp.Environment.IsProduction(). - Per-environment settings go into
appsettings.{Environment}.json.
Where to keep secrets
| Where | With what |
|---|---|
| Developer machine | User Secrets: dotnet user-secrets set |
| Server, container | environment variables, orchestrator secrets |
| Cloud | Azure Key Vault, AWS Secrets Manager, HashiCorp Vault |
Common mistake
Putting a password into appsettings.Development.json and committing it. Treat everything in the repository as public.
Takeaway: the environment picks the behavior and the settings file, and secrets live outside the repository.
How it works
- The environment name comes from
DOTNET_ENVIRONMENTorASPNETCORE_ENVIRONMENT, the--environmentargument orWebApplicationOptions.EnvironmentName. WithWebApplicationin .NET 7+,DOTNET_ENVIRONMENTwins when both variables are set. - The environment is fixed when the host is built and doesn’t change at runtime.
- In Development,
CreateBuilderloads User Secrets, andWebApplicationshows the developer exception page on its own. - User Secrets live in
secrets.jsonin the user profile:%APPDATA%\Microsoft\UserSecrets\<id>on Windows,~/.microsoft/usersecrets/<id>on Linux and macOS.<id>is theUserSecretsIdfrom the.csproj, added bydotnet user-secrets init.
When to choose what
| Option | When |
|---|---|
| User Secrets | local development only, the file isn’t encrypted |
| Environment variables | simple deployments, containers |
Secret files (AddKeyPerFile) |
Docker and Kubernetes secrets mounted into a directory |
| Key Vault and similar | cloud: managed identity, auditing, rotation |
Pitfalls
- File name casing. The Linux file system is case-sensitive: with the
Productionenvironment, a file namedappsettings.production.jsonis not picked up. launchSettings.jsonisn’t published. On a server without the variable, the environment silently becomesProduction.
Rule of thumb
The repository holds only non-secret defaults. Everything that differs between environments or is secret comes from outside.
Takeaway: the environment sets the behavior and the set of files, and at every stage secrets come from their own source, but never from git.
Under the hood
- The environment name is the
environmenthost configuration key. It is set by theDOTNET_andASPNETCORE_variables or--environment, but not byappsettings.json: files are read after the environment is chosen. - The Key Vault provider (
Azure.Extensions.AspNetCore.Configuration.Secrets) loads secrets at startup and turns--in a name into:: the secretPayments--ApiKeybecomes the keyPayments:ApiKey. It reloads them only ifReloadIntervalis set. AddKeyPerFileturns each file in a directory into a key, and__in the file name into:.
Cost
- Environment variables are visible to anyone who can see the process: they end up in dumps, in
/procon Linux and indocker inspect. - A store is a network call at startup and an external dependency: an unavailable Key Vault means the app doesn’t start.
Trade-offs
- Every
if (env.IsProduction())is code that runs for the first time in production. Differences belong in configuration, not in branches. DefaultAzureCredentialis convenient: locally it uses the developer’s account, in Azure a managed identity. For production the docs recommend a specificManagedIdentityCredential: fewer attempts and a more predictable failure.- The best secret is no secret: a managed identity for the database and cloud services removes the password from configuration entirely.
How to check in production
- At startup the host logs
Hosting environment: Production— the first thing to check when settings look “strange”. - A secret scanner in CI, such as gitleaks or GitHub secret scanning, catches passwords before merge.
Takeaway: a reliable secret is one that isn’t in the configuration: an identity instead of a password, a store instead of a file.
var builder = WebApplication.CreateBuilder(args);
// Docker/Kubernetes secrets: the file /run/secrets/Payments__ApiKey becomes the key Payments:ApiKey.// The source is added last, so it overrides environment variables too.builder.Configuration.AddKeyPerFile("/run/secrets", optional: true);
var app = builder.Build();
if (!app.Environment.IsDevelopment()){ app.UseExceptionHandler("/error"); // in Development the exception page is used instead app.UseHsts();}
app.MapGet("/env", (IHostEnvironment env, IConfiguration config) => new{ env.EnvironmentName, // "Development" HasApiKey = !string.IsNullOrEmpty(config["Payments:ApiKey"]) // never return the secret itself});
app.Run();# Locally: the secret is stored outside the repositorydotnet user-secrets initdotnet user-secrets set "Payments:ApiKey" "dev-key"
# Server or container: the colon is replaced with __export ASPNETCORE_ENVIRONMENT=Stagingexport Payments__ApiKey="..."- What is the environment if nothing is set? —
Production. - Are User Secrets encrypted? — No, it’s a plain JSON file in the user profile, for development only.
- What’s wrong with a secret in an environment variable? — It’s visible in dumps and container metadata, with no auditing or rotation.
- How does Key Vault store hierarchical keys? — With
--:Payments--ApiKeybecomesPayments:ApiKey. - What do you do with a secret that ended up in git? — Revoke and reissue it: rewriting history doesn’t help.
How do IOptions, IOptionsSnapshot and IOptionsMonitor differ, and how do you validate settings?
Section titled “How do IOptions, IOptionsSnapshot and IOptionsMonitor differ, and how do you validate settings?”What it is
The options pattern reads settings through a typed class instead of strings by key. The Payments section of appsettings.json binds to a PaymentsOptions class, and a service receives it through DI.
How to use
- Declare a class with
{ get; set; }properties. - Register it:
builder.Services.Configure<PaymentsOptions>(builder.Configuration.GetSection("Payments")). - Inject
IOptions<PaymentsOptions>and read.Value.
Which interface to use
| Interface | When it sees new values |
|---|---|
IOptions<T> |
never, read once |
IOptionsSnapshot<T> |
on the next request |
IOptionsMonitor<T> |
right after the file changes |
Common mistake
Expecting IOptions<T> to pick up an appsettings.json edit without a restart. That takes a snapshot or a monitor.
Takeaway: read settings through a typed class, and choose the interface by whether you need changes on the fly.
How it works
Configure<T>andAddOptions<T>().BindConfiguration("Payments")register not the object itself but a way to configure it.- On first access,
IOptionsFactory<T>creates an instance and applies allConfigureactions, thenPostConfigure, then validators. - The result is cached: forever for
IOptions<T>, for the request scope forIOptionsSnapshot<T>, and until the configuration changes forIOptionsMonitor<T>.
When to choose what
| Interface | Lifetime | Who it suits |
|---|---|---|
IOptions<T> |
singleton | everyone, if values don’t change without a restart |
IOptionsSnapshot<T> |
scoped | request handlers; supports named options |
IOptionsMonitor<T> |
singleton | singleton services and reacting via OnChange |
Validation
ValidateDataAnnotations()—[Required],[Range],[Url]attributes.Validate(o => ..., "message")— a rule as a lambda, for example a relation between two fields.IValidateOptions<T>— a separate class when the rule needs services.ValidateOnStart()(.NET 6+) — validation at startup instead of on the first request.
Pitfalls
IOptionsSnapshot<T>is a scoped service, so it can’t be injected into a singleton.- Nested objects aren’t validated by attributes without
[ValidateObjectMembers]and[ValidateEnumeratedItems](.NET 8+). - A typo in the section name isn’t an error: an object with default values gets bound. Only
[Required]catches it.
Rule of thumb
For every section: AddOptions<T>().BindConfiguration(...).ValidateDataAnnotations().ValidateOnStart().
Takeaway: choose the interface by the consumer’s lifetime, and run validation at startup.
Under the hood
OptionsMonitor<T>subscribes to the configuration change token. On reload it removes the instance from the cache, creates a new one through the factory and calls theOnChangelisteners.- If the new instance is invalid, the old value isn’t kept:
CurrentValuethrowsOptionsValidationExceptionon every access until the file is fixed. IOptionsSnapshot<T>creates a new instance in every scope: reflection-based binding and validation run on every request.
Cost
- A snapshot means extra allocations and CPU per request. On a hot path it’s replaced with a monitor reading
CurrentValue. ValidateDataAnnotationsand binding use reflection: they are slower and produce warnings under trimming and Native AOT.
Trade-offs
[OptionsValidator]on an emptypartialclass generates anIValidateOptions<T>at compile time. You register it as a singleton, andValidateDataAnnotationsis no longer needed.- Reloading on the fly is handy for log levels and flags. For a connection string or a pool size it’s dangerous: an object created at startup won’t forget the old value.
- Validating at startup makes a deployment fail “loudly”: the process won’t start, but it also won’t return 500 on every request.
How to check in production
With ValidateOnStart, an invalid configuration crashes the host at startup with OptionsValidationException and the full list of errors. You see it in the deployment logs and in a failed readiness probe.
Takeaway: a monitor for reloadable settings, the generator for reflection-free validation, ValidateOnStart so errors surface at deployment.
using System.ComponentModel.DataAnnotations;using Microsoft.Extensions.Options;
var builder = WebApplication.CreateBuilder(args);
builder.Services.AddOptions<PaymentsOptions>() .BindConfiguration("Payments") .ValidateDataAnnotations() .Validate(o => o.TimeoutSeconds < 120, "TimeoutSeconds must be less than 120") .ValidateOnStart(); // the error surfaces at startup, not on the first request
var app = builder.Build();
app.MapGet("/snapshot", (IOptionsSnapshot<PaymentsOptions> o) => o.Value.BaseUrl); // recomputed for this requestapp.MapGet("/monitor", (IOptionsMonitor<PaymentsOptions> o) => o.CurrentValue.TimeoutSeconds); // the current value
app.Run();
public sealed class PaymentsOptions{ [Required, Url] public string BaseUrl { get; set; } = "";
[Range(1, 300)] public int TimeoutSeconds { get; set; } = 30;}using System.ComponentModel.DataAnnotations;using Microsoft.Extensions.Options;
var builder = WebApplication.CreateBuilder(args);
builder.Services.AddOptions<PaymentsOptions>().BindConfiguration("Payments").ValidateOnStart();// Instead of ValidateDataAnnotations(): the validator is generated at compile time, no reflectionbuilder.Services.AddSingleton<IValidateOptions<PaymentsOptions>, PaymentsOptionsValidator>();
var app = builder.Build();app.Run();
[OptionsValidator]public partial class PaymentsOptionsValidator : IValidateOptions<PaymentsOptions> { }
public sealed class PaymentsOptions{ [Required, Url] public string BaseUrl { get; set; } = "";}- Can you inject
IOptionsSnapshot<T>into a singleton? — No, it’s scoped; a singleton takesIOptionsMonitor<T>. - When does validation run without
ValidateOnStart? — On the first access to.ValueorCurrentValue. - What are named options? — Several instances of one type under different names:
Get("Primary")on a snapshot or monitor. - Are nested objects validated by attributes? — No, you need
[ValidateObjectMembers]and[ValidateEnumeratedItems]. - What happens to
IOptionsMonitor<T>after an invalid file is written? — The old value isn’t kept, andCurrentValuestarts throwingOptionsValidationException.
How does logging work in ASP.NET Core, and why are messages written as templates rather than interpolated strings?
Section titled “How does logging work in ASP.NET Core, and why are messages written as templates rather than interpolated strings?”What it is
ILogger<T>— the interface for writing logs, received through DI.- Category — the logger’s name, usually the full name of class
T. Filtering is done by it. - Level — severity:
Trace,Debug,Information,Warning,Error,Critical. - Provider — where an entry goes.
CreateBuilderadds Console, Debug, EventSource and EventLog on Windows.
How to use
- Write templates:
logger.LogInformation("Order {OrderId} created", id). - Pass the exception as the first argument:
logger.LogError(ex, "Payment {PaymentId} failed", paymentId). - Set levels in the
Logging:LogLevelsection ofappsettings.json.
Common mistake
Interpolation like $"Order {id} created". The log gets a finished string with no OrderId field, and finding all entries for an order becomes hard.
Takeaway: a logger with a category, levels in configuration, messages as templates with named parameters.
How it works
ILoggerFactorycreates a logger for a category, and it hands each entry to every provider.- One rule is chosen per provider-category pair: rules for that provider beat general ones, among them the longest matching category prefix wins, and without a match
Defaultapplies. - Arguments fill the template by position, not by name. Placeholder names become property names.
- A structured provider (JSON console, Serilog, OpenTelemetry) keeps the properties separately, along with the original template.
When to choose what
| Task | Solution |
|---|---|
| Container, an agent collects logs | JSON console: AddJsonConsole() |
| Files, Seq, Elastic, many sinks | Serilog: Serilog.AspNetCore |
| Logs together with traces and metrics | export through OpenTelemetry |
Serilog
- Sink — where to write: Console, File, Seq, Elasticsearch (
Elastic.Serilog.Sinks), OpenTelemetry. - Enricher — what to add to every event:
FromLogContext,WithMachineName,WithThreadId. {@Order}destructures an object into fields,{$Order}writes itsToString().UseSerilogRequestLogging()writes one event per request instead of several from ASP.NET Core.
Pitfalls
- With Serilog, levels come from
Serilog:MinimumLevel, andLogging:LogLevelno longer applies. - The CA2254 analyzer warns when a template isn’t a constant.
Takeaway: filters work by category and provider, and a log only has structure when it uses a template with named parameters.
Under the hood
LogInformation(string message, params object?[] args)allocates an array and boxes value-type arguments before the level check. Even withDebugoff, aLogDebugcall allocates.- A template is parsed once and cached. An interpolated string is formatted at the call site, even if the entry is filtered out later.
LoggerExtensionshas no interpolated string handler overloads, so$"..."is always formatted up front.
Cost
- Every event costs CPU for formatting, allocations, network traffic and storage. Storing and indexing in Elastic often costs more than the computation.
- High-cardinality properties bloat the index. Destructure only DTOs, not EF entities with navigation properties.
Trade-offs
Microsoft.Extensions.Loggingwith export to OpenTelemetry means fewer dependencies, and logs and traces share one protocol.- Serilog has richer sinks and formatting, but it’s a second pipeline with its own level configuration. It can write to OTLP too:
Serilog.Sinks.OpenTelemetry. - A sink that writes to the network synchronously slows requests down. Slow sinks are wrapped in
Serilog.Sinks.Async, and you watch for buffer overflow.
How to check in production
- Aggregating in the log store by message template and category shows the “noisiest” lines of code.
- Logging allocations show up in an allocation profiler (dotnet-trace) and in BenchmarkDotNet with
[MemoryDiagnoser].
Takeaway: a template is both structure and a grouping key, while interpolation pays for formatting even when the log isn’t needed.
{ "Logging": { "LogLevel": { "Default": "Information", "Microsoft.AspNetCore": "Warning", "Orders": "Debug" }, "Console": { "FormatterName": "json" } }}var builder = WebApplication.CreateBuilder(args);var app = builder.Build();
app.MapPost("/orders/{id:int}", (int id, ILoggerFactory factory) =>{ ILogger log = factory.CreateLogger("Orders.Api"); // prefix "Orders" → Debug level
// Good: OrderId and Total are separate fields, one template for all orders log.LogInformation("Order {OrderId} created, total {Total}", id, 99.5m);
// Bad: a finished string without fields, formatted even when the level is off log.LogInformation($"Order {id} created, total {99.5m}");
log.LogDebug("Reserved stock for {OrderId}", id); // logged: Debug is on for "Orders" return Results.Created($"/orders/{id}", null);});
app.Run();using Serilog;
Log.Logger = new LoggerConfiguration() .WriteTo.Console() .CreateBootstrapLogger(); // catches startup errors before the host is built
try{ var builder = WebApplication.CreateBuilder(args);
builder.Services.AddSerilog((services, lc) => lc .ReadFrom.Configuration(builder.Configuration) // levels and sinks from the "Serilog" section .ReadFrom.Services(services) .Enrich.FromLogContext()); // properties from scopes and LogContext
var app = builder.Build(); app.UseSerilogRequestLogging(); // one event per request app.MapGet("/", () => "Hello"); app.Run();}catch (Exception ex){ Log.Fatal(ex, "Host terminated unexpectedly");}finally{ Log.CloseAndFlush(); // flush sink buffers before exit}{ "Serilog": { "MinimumLevel": { "Default": "Information", "Override": { "Microsoft.AspNetCore": "Warning" } }, "WriteTo": [ { "Name": "Console" }, { "Name": "File", "Args": { "path": "logs/app-.log", "rollingInterval": "Day" } } ] }}- How is the level chosen for the
Microsoft.AspNetCore.Routingcategory? — A provider rule beats a general one, then the longest matching category prefix wins. - Why are arguments matched by position, not by name? — The template is parsed positionally; names only become property names.
- How does
{@Order}differ from{Order}in Serilog? —@destructures the object into fields; without it,ToString()is written. - Why did the levels from
Loggingstop working after adding Serilog? — Serilog does the filtering itself; levels are set inSerilog:MinimumLevel. - How do you log an exception properly? — Pass it as the first argument,
LogError(ex, ...), instead of puttingex.Messageinto the text.
How do you make logging fast and observable: LoggerMessage, scopes, correlation and OpenTelemetry?
Section titled “How do you make logging fast and observable: LoggerMessage, scopes, correlation and OpenTelemetry?”What it is
[LoggerMessage]— an attribute on apartialmethod. The generator writes the method body: a level check and an entry without extra allocations.- Scope — a
using (logger.BeginScope(...))block whose properties are added to every entry inside it. - Correlation — a shared
TraceIdon all entries of one request, even when the request passes through several services. - OpenTelemetry — a standard and libraries for three signals: logs, metrics and traces.
How to use
- Declare frequent messages as methods with
[LoggerMessage]. - Put the order or user ID into a scope once instead of into every message.
- Add OpenTelemetry and OTLP export to your backend.
Common mistake
Inventing your own CorrelationId when a TraceId already exists. You end up with two unrelated identifiers.
Takeaway: [LoggerMessage] for speed, scopes and TraceId for connected logs.
How it works
- The generator creates a state struct for the method and calls
logger.Logonly afterIsEnabled. Value-type arguments aren’t boxed. BeginScopepushes state onto a stack held in anAsyncLocal, so a scope survivesawait. The console provider prints scopes withIncludeScopes.- ASP.NET Core starts an
Activityand a scope withRequestIdandRequestPathfor each request. The logger addsTraceId,SpanIdandParentIdof the currentActivity. HttpClientsends thetraceparentheader, and the next service continues the same trace.
When to choose what
| Signal | The question it answers |
|---|---|
Metrics (Meter) |
how many and how fast: RPS, latency, error rate |
Traces (Activity) |
where a specific request spends its time |
Logs (ILogger) |
what exactly happened and with which data |
Pitfalls
- A
[LoggerMessage]method must bepartialand returnvoid. The exception is passed as a parameter and not mentioned in the template. ActivitySource.StartActivityreturnsnullwhen nobody listens to the source, so you access it throughactivity?..- A scope with a dictionary allocates on every call, so you don’t open one in a hot loop.
Rule of thumb
Metrics for alerts, traces for finding the bottleneck, logs for details; TraceId ties them together.
Takeaway: the generator removes the cost of disabled logs, and Activity gives logs and traces a shared TraceId.
Under the hood
- The generated method passes a strongly typed state struct to
ILogger.Log. If the level is off, neither an array nor a string is created. Activity.Currentlives in anAsyncLocaland flows throughawaitwithExecutionContext. Thetraceparentheader carries thetrace-id, theparent-idand the sampled flag.- In .NET 10 the default propagator is W3C: baggage goes in the
baggageheader instead of the non-standardCorrelation-Context.
Cost
- Telemetry costs CPU for serialization, memory for exporter buffers and network traffic. The OpenTelemetry exporter batches in the background and drops data when its queue is full rather than blocking requests.
- Storage is paid by the gigabyte. Traces are sampled (
ParentBasedSamplerandTraceIdRatioBasedSampler),Debugis off in production, and detailed logs are buffered and emitted only on error (AddPerIncomingRequestBuffer, .NET 9+).
Trade-offs
- The Elastic APM agent instruments everything itself but ties you to the vendor. OpenTelemetry is a standard: the backend changes with the export address, and Elastic accepts OTLP and ships its own EDOT .NET distribution.
- Since 1.23, the Elastic APM .NET agent picks up
ActivitySourcespans itself, so manualActivityinstrumentation doesn’t tie you to the agent. - High-cardinality metric tags (user ID, URLs with parameters) blow up the number of time series. Such values belong in traces and logs.
How to check in production
dotnet-counters monitor --counters Microsoft.AspNetCore.Hosting,System.Runtimeshows metrics of a running process without an SDK or a restart.dotnet-tracewith theMicrosoft-Extensions-Loggingprovider captures a process’s logs even when the log store is unavailable.
Takeaway: a fast log does no work before the level check, and observability is a shared TraceId plus a budget for telemetry volume.
using System.Diagnostics;
var builder = WebApplication.CreateBuilder(args);builder.Logging.AddJsonConsole(o => o.IncludeScopes = true); // scopes and TraceId in every entry
var app = builder.Build();var source = new ActivitySource("Orders");
app.MapPost("/orders/{id:int}", (int id, ILogger<Program> logger) =>{ using var activity = source.StartActivity("ProcessOrder"); // a span; null if nobody listens activity?.SetTag("order.id", id);
using (logger.BeginScope(new Dictionary<string, object> { ["OrderId"] = id })) { OrderLog.Processing(logger, id, "card"); // the level is checked before any work } return Results.Accepted();});
app.Run();
static partial class OrderLog{ [LoggerMessage(EventId = 1001, Level = LogLevel.Information, Message = "Processing order {OrderId} via {Method}")] public static partial void Processing(ILogger logger, int orderId, string method);}// NuGet: OpenTelemetry.Extensions.Hosting, OpenTelemetry.Instrumentation.AspNetCore, OpenTelemetry.Instrumentation.Http, OpenTelemetry.Exporter.OpenTelemetryProtocolusing OpenTelemetry;using OpenTelemetry.Metrics;using OpenTelemetry.Resources;using OpenTelemetry.Trace;
var builder = WebApplication.CreateBuilder(args);
builder.Services.AddOpenTelemetry() .ConfigureResource(r => r.AddService("orders-api")) .WithTracing(t => t .AddSource("Orders") // our ActivitySource .AddAspNetCoreInstrumentation() .AddHttpClientInstrumentation()) .WithMetrics(m => m .AddMeter("Microsoft.AspNetCore.Hosting") // built-in ASP.NET Core metrics (.NET 8+) .AddMeter("System.Runtime")) // GC and thread pool (.NET 9+) .WithLogging() // ILogger → OTLP together with TraceId and SpanId .UseOtlpExporter(); // address from OTEL_EXPORTER_OTLP_ENDPOINT
var app = builder.Build();app.MapGet("/", () => "Hello");app.Run();Switching the backend (Elastic, Jaeger, Grafana) means changing the OTLP address; the code stays the same.
- Why is
[LoggerMessage]faster thanLogInformation? — The level is checked before any work, there’s noparamsarray or boxing, and the template is parsed at compile time. - Where does
TraceIdin your logs come from if you never wrote it? — The logger addsTraceId,SpanIdandParentIdof the currentActivity(ActivityTrackingOptions). - Why did
StartActivityreturnnull? — Nobody listens to the source: OpenTelemetry isn’t subscribed to it throughAddSource. - How does head-based sampling differ from tail-based? — Head decides at the start of a trace in the app; tail decides after it completes, in the collector, and can keep every trace with errors.
- How do you link Serilog logs to traces? — Serilog stores
TraceIdandSpanIdof the activeActivityby itself; you just need to output them in the sink.