How to run work outside an HTTP request and how to cache data and responses: hosted services and Worker Service, Hangfire and Quartz.NET, graceful shutdown and background failures, in-memory and distributed cache, output cache and HybridCache, cache stampede and invalidation. In interviews this topic tests whether you understand the guarantees: what survives a restart, what runs twice and what goes stale.
IHostedService — a service with two methods: the host calls StartAsync when the app starts and StopAsync when it stops.
BackgroundService — a base class on top of it. You only override ExecuteAsync(CancellationToken stoppingToken), the method with the background loop.
Why
To do work outside HTTP requests: read a queue, sync data on a timer, warm up a cache.
How to use
Derive from BackgroundService and run a loop until stoppingToken is cancelled.
Register the service: builder.Services.AddHostedService<Worker>().
For a process without HTTP, use the dotnet new worker template. Such a worker runs as a Windows service (AddWindowsService()) or under systemd (AddSystemd()).
Common mistake
Not passing stoppingToken to Task.Delay and other calls. Then app shutdown waits for the timeout.
Takeaway:BackgroundService plus AddHostedService is the standard way to run background work.
How it works
AddHostedService<T>() registers the service as a singletonIHostedService.
On startup the host calls StartAsync on every service, one at a time, in registration order. In ASP.NET Core this happens before the server starts.
BackgroundService.StartAsync starts ExecuteAsync and returns immediately. The task lives until stoppingToken is cancelled.
On shutdown the host calls StopAsync in reverse order. BackgroundService cancels the token and waits for ExecuteAsync, but no longer than the shutdown timeout.
When to choose what
Task
Choice
Endless loop: queue, timer
BackgroundService
Short initialization before serving requests
IHostedService with work in StartAsync
Background process without HTTP
Worker Service: Host.CreateApplicationBuilder
Pitfalls
A hosted service is a singleton, so you cannot inject DbContext or other scoped services into its constructor. Create a scope per iteration with IServiceScopeFactory.CreateAsyncScope() — see Dependency injection for details.
For periodic work use PeriodicTimer (.NET 6+). Unlike System.Threading.Timer, its iterations never overlap.
Every app instance runs its own copy of the worker. There is no coordination between them.
Rule of thumb
Long loop — in ExecuteAsync, short initialization — in StartAsync, a new scope — per unit of work.
Takeaway: a hosted service lives as long as the host and is driven by it through StartAsync, StopAsync and the stopping token.
Under the hood
AddHostedService registers the service via TryAddEnumerable: registering the same type twice does not create a second instance.
Before .NET 10, the synchronous part of ExecuteAsync up to the first await ran inside StartAsync and delayed the start of other services and the server. Since .NET 10 BackgroundService runs the whole method via Task.Run.
AddWindowsService() and AddSystemd() do not touch service code; they replace IHostLifetime. A Windows service also gets AppContext.BaseDirectory as its content root, because its current directory is System32.
Cost
No persistence: in-memory work is lost on deploy, restart or crash.
When you scale out, every instance does the same work: two pods, two nightly reports.
Background work shares the thread pool and CPU with requests.
Trade-offs
Option
Upside
Downside
Worker in the web process
one deployment
a web restart kills jobs
Separate Worker Service
own releases and scaling
one more process
A single executor across the whole farm needs a lock in shared storage or a scheduler with clustering.
How to check in production
Log the start and end of each iteration and expose the time of the last iteration in a health check. A stuck worker shows up as a timestamp that stopped moving.
Takeaway: a hosted service is a simple in-process mechanism; it gives no delivery guarantees, no scheduling and no single-executor guarantee across a farm.
Example
var builder =WebApplication.CreateBuilder(args);
builder.Services.AddHostedService<HeartbeatWorker>(); // singleton, starts with the host
while (awaittimer.WaitForNextTickAsync(stoppingToken)) // cancelled on shutdown
{
logger.LogInformation("Heartbeat at {Time}", DateTimeOffset.Now);
}
}
}
Likely follow-ups
How does BackgroundService differ from IHostedService? — It is a base class: it starts ExecuteAsync itself, and in StopAsync it cancels the token and waits for the task.
Can you inject DbContext into a worker’s constructor? — No, the worker is a singleton; create a scope via IServiceScopeFactory per iteration.
When does StartAsync run relative to Kestrel? — Before the server starts, sequentially for all services.
How do you run a worker as a Windows service? — The Microsoft.Extensions.Hosting.WindowsServices package, AddWindowsService() and registration with sc.exe create.
What happens with two app instances? — Two workers do the same job; remove duplicates with a lock or a clustered scheduler.
Register the cache: AddMemoryCache() or, for example, AddStackExchangeRedisCache(...) from the Microsoft.Extensions.Caching.StackExchangeRedis package.
Inject the interface and read through GetOrCreateAsync, or GetStringAsync and SetStringAsync.
Set a lifetime:
absolute (AbsoluteExpirationRelativeToNow) — the entry lives no longer than N minutes;
sliding (SlidingExpiration) — the entry is removed if nobody read it for N minutes, and every read extends it.
Common mistake
Setting only sliding expiration. A popular entry is read all the time and serves stale data for ages.
Takeaway: memory is fast, but every server has its own copy; a distributed cache is shared, but goes over the network.
How it works
MemoryCache is a thread-safe in-process dictionary. Values are stored by reference: change an object after reading it and you change it for everyone.
IDistributedCache only knows byte[]; serialization is up to you. Refresh extends the sliding window without reading the value.
AddDistributedMemoryCache() is an in-process implementation of IDistributedCache. It is for development and tests and is not actually distributed.
When to choose what
Situation
Cache
Single instance, reference data, settings
IMemoryCache
Farm, data must match across nodes
IDistributedCache (Redis)
Need both memory speed and a shared cache
L1 in memory on top of L2 in Redis
Pitfalls
IMemoryCache is not trimmed under memory pressure. Bound it with SizeLimit on a dedicated MemoryCache instance: with a limit, everyone who writes to the cache must specify a size.
Mutating a cached object changes it for every reader. Cache immutable data.
Keys built from user inputbloat the cache — a DoS vector.
Rule of thumb
The code must work without the cache: a cache is an optimization, not a data store.
Takeaway:IMemoryCache is for data that need not be consistent across nodes, IDistributedCache is for data the whole farm shares.
Under the hood
Expiration is lazy. An expired entry is never returned on read; it is removed during cache accesses and background scans no more often than ExpirationScanFrequency (once a minute by default).
With SizeLimit set, an entry without Size throws InvalidOperationException. When the cache overflows it compacts itself: expired entries first, then by priority and recency of use.
IDistributedCache stores bytes together with expiration metadata. A sliding window is extended on read, which means a write to the store on every hit.
Cost
L1 multiplies memory by the number of nodes. Large long-lived objects settle in generation 2 and the LOH.
L2 costs a network call and serialization per hit. A Redis outage must not take the app down: short timeouts and a fallback to the source.
Trade-offs
L1 on different nodes diverges for the lifetime of an entry. A short L1 lifetime or invalidation messages narrow that window.
A two-level scheme gives memory speed plus a shared L2, but makes invalidation harder.
How to check in production
MemoryCacheOptions.TrackStatistics = true and GetCurrentStatistics() (.NET 7+) report hits, misses and entry count.
Memory growth shows up in dotnet-counters as heap size, and what the cache actually holds shows up in dotnet-gcdump.
Takeaway: a local cache is speed without consistency, a distributed cache is consistency at the cost of the network; size and lifetimes are always the developer’s job.
Example
usingMicrosoft.Extensions.Caching.Distributed;
usingMicrosoft.Extensions.Caching.Memory;
var builder =WebApplication.CreateBuilder(args);
builder.Services.AddMemoryCache();
builder.Services.AddDistributedMemoryCache(); // in production: AddStackExchangeRedisCache(...)
Hangfire runs ordinary methods in the background: BackgroundJob.Enqueue(() => SendEmail(id)). The job is saved to the database, executed, retried on failure, and everything is visible in a web dashboard.
Quartz.NET is a scheduler: a job (IJob) plus a trigger that decides when to run it, for example on a cron schedule.
Why
BackgroundService lives in process memory. Restart the app and you lose queued work and miss scheduled runs.
How to use
Need
Tool
Loop, polling, timer
BackgroundService
Send an email in the background with retries
Hangfire
Nightly report at 02:00
Hangfire RecurringJob or Quartz
Common mistake
Passing an EF entity or a large object to a Hangfire job. Arguments are serialized to JSON and stored in the database, so pass an id.
Takeaway: if you need reliability or a schedule, use a scheduler; for a simple loop BackgroundService is enough.
How it works
Hangfire. The client serializes the method call — type, method, arguments — into storage. The Hangfire server (a hosted service with a pool of workers) picks jobs from the queue. The AutomaticRetry filter retries a failed job 10 times by default with growing delays.
Quartz.NET. Jobs and triggers live in a JobStore: RAMJobStore in memory by default, AdoJobStore in a database for durability. Missed runs, for example while the app was down, are handled by the trigger’s misfire policy.
When to choose what
Criterion
Hangfire
Quartz.NET
Model
queue of method calls
jobs and triggers
Storage
always persistent
in memory or AdoJobStore
Retries
AutomaticRetry out of the box
manual
Scheduling
cron
cron with seconds, calendars
Multiple nodes
shared storage
clustered AdoJobStore
Dashboard
built in
Quartz.Dashboard package
Pitfalls
Hangfire stores type and method names: renaming a method or changing its signature breaks jobs already sitting in the queue.
After a worker crash a job runs again, so make it idempotent.
The Hangfire dashboard only allows local requests by default; production needs your own authorization filter.
Rule of thumb
Persistence, retries or “one executor per farm” — an external scheduler. Everything else — BackgroundService.
Takeaway: Hangfire is a reliable queue of calls with retries, Quartz is precise scheduling with clustering.
Under the hood
Hangfire with SQL Server fetches a job so that other servers cannot see it. If the server dies mid-job, the job goes back to the queue after a timeout and runs again — hence the at-least-once guarantee.
Quartz cluster nodes compete for triggers through row locks in the database and check in every 7.5 seconds. A node whose clock runs fast can declare a healthy peer dead, so clocks must be synchronized.
Cost
Constant load on the database plus its own table schema with migrations.
Job arguments are a contract: their serialized form outlives the version of the code that produced it.
Running twice is normal, not an incident. Build idempotency into every job.
Trade-offs
Redis storage and batches in Hangfire come with the paid Pro edition.
Quartz 4.0 (September 2026) runs only on .NET 10 and changed the API: IJob.Execute returns ValueTask and takes a CancellationToken, and the database schema needs a migration. The 3.x line is still maintained for .NET Framework.
Passing jobs between services — a message broker. Periodically running a separate process — a Kubernetes CronJob.
How to check in production
Watch the number of jobs in the Failed and Retrying states and queue lengths in the dashboard. In Quartz, alert on the message is still active but was recovered by another instance — a sign of clock skew in the cluster.
Takeaway: a scheduler gives persistence and retries at the cost of database load, versioned arguments and mandatory idempotency.
Response caching: [ResponseCache(Duration = 60)] sets Cache-Control: public,max-age=60. Server-side caching comes from AddResponseCaching() and UseResponseCaching().
Output caching: AddOutputCache(), UseOutputCache() and .CacheOutput() on an endpoint.
HybridCache: the Microsoft.Extensions.Caching.Hybrid package, AddHybridCache() and GetOrCreateAsync(key, factory).
Common mistake
Caching responses for authenticated users: one user gets another’s data. Output cache does not cache such responses by default — do not turn that rule off without varying the key by user.
Takeaway: headers are for clients and CDNs, output cache is for responses on the server, HybridCache is for data.
How it works
Response caching middleware follows RFC 9111: it caches GET and HEAD with status 200, without Authorization and Set-Cookie, and honours the request’s Cache-Control. A browser sending max-age=0 bypasses the cache.
Output caching ignores client headers. By default it caches only 200 responses to GET and HEAD, without cookies and without authentication, for 60 seconds. The key is the whole URL, tuned with SetVaryByQuery and SetVaryByHeader.
HybridCache: GetOrCreateAsync checks L1 (MemoryCache), then L2 (the registered IDistributedCache), and only then calls the factory — once per key per app instance. Values are serialized with System.Text.Json.
When to choose what
Task
Choice
Static content and public GETs for a CDN
Cache-Control, [ResponseCache]
Whole pages and API responses
output cache
Results of database and external API calls
HybridCache
Pitfalls
Put UseOutputCache after UseCors, UseAuthentication and UseAuthorization, otherwise you may serve a response cached for another user.
For output cache on several nodes use AddStackExchangeRedisOutputCache (.NET 8+), not IDistributedCache: tags need atomicity.
By default HybridCache deserializes a new copy on every read. Instance reuse is enabled by sealed plus [ImmutableObject(true)] on the type.
Rule of thumb
First decide what you cache — a response or data — then where the copy lives and who clears it.
Takeaway: the server controls output cache and HybridCache; the client controls response caching.
Under the hood
Output cache keeps responses in IOutputCacheStore: in memory with a 100 MB limit by default, while the Redis store supports tags atomically. Revalidation via ETag and If-Modified-Since returns 304 without a body.
HybridCache coalesces concurrent calls for a key within an instance. The factory’s token is cancelled only when every waiting caller has cancelled.
HybridCache tags are invalidation timestamps: on read, an entry older than the timestamp counts as a miss.
Cost
HybridCache: serialization on every L2 write and read. Entries larger than MaximumPayloadBytes (1 MB by default) are not cached.
Output cache without Redis keeps a separate copy on every node. SetVaryByQuery("*") multiplies the number of entries by the number of parameter combinations.
Trade-offs
Freshness versus load: a CDN max-age cannot be recalled. A common setup is a short max-age for clients plus output cache with tag eviction on the server.
HybridCache has no built-in way to broadcast invalidations between nodes (a backplane). FusionCache, which implements the same abstraction, provides one.
How to check in production
Look at the Age and Cache-Control headers in responses, count factory calls and database queries, and compare P95 latency before and after enabling the cache.
Takeaway: pick the layer by who must be able to clear the copy: a CDN — nobody, output cache — the server by tag, HybridCache — code by key or tag.
Output cache and Cache-Control
usingMicrosoft.AspNetCore.OutputCaching;
var builder =WebApplication.CreateBuilder(args);
builder.Services.AddOutputCache(o =>
o.AddPolicy("products", p =>p.Expire(TimeSpan.FromMinutes(5)).Tag("products")));
var app =builder.Build();
app.UseOutputCache();
// The whole response is stored on the server; the client's Cache-Control has no effect
Graceful shutdown means a background task finishes its current work and exits on its own instead of being cut off mid-operation.
How it works
The app receives a stop signal: Ctrl+C, a service stop, docker stop.
The host cancels the stoppingToken that ExecuteAsync received.
The code notices the cancellation — through an await with that token or an IsCancellationRequested check — and exits.
The host waits for it, but no longer than the timeout: 30 seconds by default.
What about exceptions
If ExecuteAsync throws, the host logs the exception and stops the app. So the error of a single iteration is caught inside the loop, logged, and the loop continues.
Common mistake
while (true) and Task.Delay without the token. The app hangs until the timeout, and the work is cut off.
Takeaway: pass stoppingToken to every call and catch per-iteration errors inside the loop.
How it works
A stop signal raises ApplicationStopping, and the host calls StopAsync on services in reverse registration order.
BackgroundService.StopAsync cancels the token and waits for ExecuteAsync or for the stop token to fire, whichever comes first. That token fires after ShutdownTimeout.
When time runs out, the host carries on shutting down, and unfinished work dies with the process.
The host logs an exception from ExecuteAsync as BackgroundService failed. An OperationCanceledException during shutdown is not treated as an error.
When to choose what
Situation
Choice
An iteration may fail, the service must keep running
try/catch inside the loop
The app is pointless without the service
let the exception stop the host
Shutdown needs more time
HostOptions.ShutdownTimeout
Pitfalls
BackgroundServiceExceptionBehavior.Ignore turns a failed service into a zombie: the process is alive, but no work happens.
StartAsync runs sequentially and before the server starts, so heavy work there delays startup.
A timer that starts a new iteration without waiting for the previous one leaves a tail of unfinished operations on shutdown.
Rule of thumb
Every await in background work gets stoppingToken, per-iteration errors are caught in the loop, and a failure of the whole service is a signal to stop the host.
Takeaway: the host asks for a stop with a token and waits for a limited time; whatever has not finished gets cut off.
Under the hood
The host awaits each BackgroundService task separately. On failure it logs and, with StopHost, calls StopApplication().
In .NET 11 (due in November 2026), RunAsync and StopAsync throw the service’s exception after such a stop, and the exit code becomes non-zero. Before that the app exited with code 0.
IHostedLifecycleService (.NET 8) adds StartingAsync, StartedAsync, StoppingAsync and StoppedAsync. The stopping methods count against the same ShutdownTimeout.
Cost
ShutdownTimeout is one budget for everyone: Kestrel drains requests, and services stop one after another unless ServicesStopConcurrently is on.
The orchestrator’s window is stricter: Kubernetes gives 30 seconds before SIGKILL, docker stop 10, systemd 90.
Trade-offs
A Channel<T> queue loses items on shutdown and on crash. The options are to drain it within the budget or keep work outside the process (Hangfire, a broker, an outbox).
A bounded channel gives backpressure: when it is full, WriteAsync waits, and requests slow down instead of memory growing.
For a Windows service the docs recommend Environment.Exit(1) after a fatal error so that service recovery kicks in. The price: finally blocks and graceful shutdown do not run.
How to check in production
Search the logs for BackgroundService failed and measure the time from Application is shutting down to exit. On a staging environment, send SIGTERM under load and check that in-flight work completes.
Takeaway: graceful shutdown means a token in every call, a budget smaller than the orchestrator’s window, and work that survives being cut off.
Example
usingSystem.Threading.Channels;
var builder =WebApplication.CreateBuilder(args);
builder.Services.Configure<HostOptions>(o =>
{
o.ShutdownTimeout=TimeSpan.FromSeconds(20); // below the orchestrator's window (K8s: 30 s)
o.BackgroundServiceExceptionBehavior=BackgroundServiceExceptionBehavior.StopHost; // the default
});
builder.Services.AddSingleton(Channel.CreateBounded<string>(100)); // when full, WriteAsync waits
builder.Services.AddHostedService<EmailSender>();
var app =builder.Build();
app.MapPost("/emails", async (string to, Channel<string> queue, CancellationToken ct) =>
// On shutdown reading stops, and unread addresses are lost
awaitforeach (var to inqueue.Reader.ReadAllAsync(stoppingToken))
{
try
{
awaitTask.Delay(200, CancellationToken.None); // finish the email we started
}
catch (Exception ex)
{
logger.LogError(ex, "Failed to send email to {To}", to); // one failure doesn't bring down the host
}
}
}
}
Likely follow-ups
How long does the host wait for background services by default? — 30 seconds since .NET 6, 5 before; set it with HostOptions.ShutdownTimeout.
Why doesn’t a Windows service restart after the worker fails? — StopHost stops the host cleanly with exit code 0; you need Environment.Exit(1) or .NET 11.
What happens to items in a Channel<T> on shutdown? — Unprocessed ones are lost; drain within the budget or keep the queue outside the process.
Why IHostedLifecycleService? — Code before and after StartAsync and StopAsync of all services: migrations before startup, draining before shutdown.
Will a console app without a host handle SIGTERM on .NET 10? — No: the runtime no longer installs a handler; the host installs one through ConsoleLifetime.
A cache stampede is a mass miss: an entry expired, and every request goes to the database at once. The database gets hundreds of times its usual load and may fall over.
How to protect against it
One request per key, the others wait for its result. That is how HybridCache.GetOrCreateAsync and output cache work.
Different lifetimes: add random seconds to the expiration so keys do not expire at the same moment.
How to update data
Changed data in the database — delete the key from the cache (RemoveAsync). The next read loads the fresh version.
Common mistake
Assuming IMemoryCache.GetOrCreateAsync calls the factory once. With concurrent misses several requests will call it.
Takeaway: on a miss one request goes to the source, and after a write the key is deleted.
How it works
Cache-aside: read the cache; on a miss load from the source and store with a lifetime.
Write: commit to the database, then delete the key. Deletion is idempotent and never stores a value that lost a race.
Request coalescing (single-flight): a task for the in-flight load is kept per key, and other misses await it.
When to choose what
Pattern
When
Cache-aside
the default: the cache is optional, load lazily
Write-through
writes go to cache and database synchronously; reads right after writes are common
Write-behind
write to cache, flush to the database later in batches; some risk of loss is acceptable
Pitfalls
Data loaded at startup with the same lifetime expires all at once. Add jitter.
Deleting the key before the transaction commits: a concurrent read puts the old value back.
Non-existent keys hit the database every time. Cache “not found” too, with a short lifetime.
Rule of thumb
Always a lifetime, delete the key after commit, coalesce requests for expensive keys.
Takeaway: cache-aside with delete-after-write and jittered lifetimes solves most freshness and load problems.
Under the hood
HybridCache coalesces calls only within an instance: five nodes mean up to five loads. Output cache resource locking is local too.
A distributed lock (for example SET NX PX in Redis) brings loads down to one but needs a timeout: the holder may die, and the lock may expire mid-work.
Refreshing early (refresh-ahead) and serving the stale value while reloading (stale-while-revalidate) take misses off the hot path. In FusionCache these are eager refresh and fail-safe.
Cost
Waiting callers share the latency of one load. If it fails, they all get the error — or, with fail-safe, the stale value.
Trade-offs
The race: request A misses and reads the old value from the database, B writes the new value and deletes the key, A stores the old value in the cache. The lifetime bounds this window; short lifetimes, versioned keys (product:42:v7) and a delayed second delete narrow it.
L1 on other nodes is cleared with an event (Redis pub/sub, a bus, CDC or an outbox) or a short LocalCacheExpiration.
Groups of entries are cleared with tags: RemoveByTagAsync in HybridCache, EvictByTagAsync in output cache.
How to check in production
Track the hit ratio and the number of factory calls per key. Database latency spikes that line up with lifetime boundaries are a sign of synchronized expiration. Run load tests with a cold cache too — after a deploy or a Redis flush.
Takeaway: a stampede is tamed by request coalescing and jittered lifetimes, and consistency is bounded by lifetimes, delete-after-commit and invalidation events.
Example
usingSystem.Collections.Concurrent;
usingMicrosoft.Extensions.Caching.Memory;
var cache =new MemoryCache(new MemoryCacheOptions());
var inFlight =new ConcurrentDictionary<int, Lazy<Task<decimal>>>();
int dbCalls =0;
async Task<decimal> GetPriceAsync(int id)
{
if (cache.TryGetValue(id, outdecimal price)) return price; // hit
// Single-flight: every miss for the key awaits one load
var load =inFlight.GetOrAdd(id, key =>new Lazy<Task<decimal>>(async () =>
{
Interlocked.Increment(ref dbCalls);
awaitTask.Delay(100); // "database query"
var ttl =TimeSpan.FromMinutes(5) +TimeSpan.FromSeconds(Random.Shared.Next(60)); // jitter
returncache.Set(key, key *10m, ttl);
}));
try { returnawaitload.Value; }
finally { inFlight.TryRemove(KeyValuePair.Create(id, load)); } // remove only our own load