When a hot key expires: stopping cache stampedes with FILLGET

Cache-aside looks harmless on a whiteboard. Read the key; on a miss, load from the database and write it back. The trouble starts when the key is popular. It expires, and every request that arrives in the next few hundred milliseconds sees the same miss and runs the same expensive query. The database that the cache was protecting takes the full load at the worst possible moment.

People fix this with a lock around the refill, a random jitter on the TTL, a background refresher, or all three. Each fix adds a round trip or a moving part. I wanted the cache itself to answer the real question: should this caller load the value, or not?

1: caller rebuilds an expired key; the rest wait or read the stale copy
5: possible answers, so the client never has to guess
SWR: stale-while-revalidate window, set per write

One command, five answers

FILLGET reads the key and, if it is missing, makes the caller the filler or tells it that someone else already is:

FILLGET product:42 5000
# 1) FILL  2) 1    -> missing: you rebuild it, here is your fill token
FILLGET product:42 5000
# 1) BUSY  2) (nil) -> someone else is rebuilding it
FILLSET product:42 "{...}" 60000 1 SWR 30000
# 1                -> stored with fill token 1, fresh for 60 s
FILLGET product:42 5000
# 1) HIT   2) "{...}"

use std::time::Duration;
use ruvio_client::{Client, FillOutcome};

let mut db = Client::connect("127.0.0.1", 6379)?;
let outcome = db.fill_get_wait("product:42", Duration::from_millis(5000), Duration::from_millis(200))?;
// Hit(value)  -> served from memory      Fill(token)            -> you refill it
// Busy        -> someone else is filling Stale { value, token } -> inside a stale-while-revalidate window
if let FillOutcome::Fill(token) | FillOutcome::Stale { token: Some(token), .. } = outcome {
    let stored = db.fill_set("product:42", "{...}", Duration::from_secs(60), &token, Some(Duration::from_secs(30)))?;
    // true -> stored (false if the token is no longer live)
}

await using var connection = await RuvioConnection.ConnectAsync("127.0.0.1", 6379);
var db = await connection.GetDefaultDatabaseAsync();

// One caller builds the value; the others poll with short FILLGET calls, so the connection is never held.
string html = await db.GetOrCreateAsync("product:42", TimeSpan.FromSeconds(60),
    async ct => await LoadProductHtmlAsync(42, ct));

c, err := ruvio.Connect(ctx, ruvio.Options{Addr: "127.0.0.1:6379"})
if err != nil { return err }

f, err := c.FillGetWait(ctx, "product:42", 5*time.Second, 200*time.Millisecond)
if err != nil { return err }
// f.State: ruvio.FillHit (f.Value), FillFill (f.Token), FillBusy, or FillStale (f.Value, f.Token)
if f.Token != "" { // you refill it
    stored, err := c.FillSet(ctx, "product:42", "{...}", time.Minute, f.Token, 30*time.Second)
    if err != nil { return err }
    fmt.Println(stored) // true -> stored (false if the token is no longer live)
}

var c = RuvioClient.connect("127.0.0.1", 6379);

var fill = c.fillgetWait("product:42", 5000, 200);
// fill.state(): HIT (fill.value()), FILL (fill.token()), BUSY, or STALE (fill.value(), fill.token())
if (fill.token() != null) { // you refill it
    Boolean stored = c.fillset("product:42", "{...}", 60_000, fill.token(), 30_000);
    // true -> stored (false if the token is no longer live)
}

import ruvio

r = ruvio.Ruvio("127.0.0.1", 6379)
fill = r.fillget("product:42", 5000, wait_ms=200)
# FillResult(state, value, token): state is HIT, FILL, BUSY or STALE
if fill.token is not None:  # you refill it
    stored = r.fillset("product:42", "{...}", 60_000, fill.token, swr_ms=30_000)
    # True -> stored (False if the token is no longer live)

import { createClient } from 'ruvio-client';

const c = createClient({ host: '127.0.0.1', port: 6379 });
const fill = await c.fillget('product:42', 5000, 200);
// { state: 'hit' | 'fill' | 'busy' | 'stale', value, token }
if (fill.token !== null) { // you refill it
  const stored = await c.fillset('product:42', '{...}', 60_000, fill.token, 30_000);
  // true -> stored (false if the token is no longer live)
}

builder.Services.AddRuvioClient(builder.Configuration.GetSection("Ruvio"));
builder.Services.AddRuvioDistributedCache(options => options.InstanceName = "orders");

// One caller across every instance runs the factory; the others receive its result.
var entry = new DistributedCacheEntryOptions { AbsoluteExpirationRelativeToNow = TimeSpan.FromMinutes(10) };

// A plain string: no byte[] conversion on your side.
string page = await cache.GetOrCreateStringAsync("invoice:42", token => RenderAsync(token), entry);

// A model stored as JSON; pass your JsonSerializerOptions, or a source-generated JsonTypeInfo<T> for AOT.
Invoice invoice = await cache.GetOrCreateJsonAsync(
    "invoice-model:42", token => LoadInvoiceAsync(token), AppJson.Default.Invoice, entry);

// Already have a JSON string? Store it as a string.
await cache.SetStringAsync("invoice-json:42", json, entry);

// Serve the last good value for an hour while one caller refreshes it, and tag it.
var resilient = new RuvioCacheEntryOptions
{
    AbsoluteExpirationRelativeToNow = TimeSpan.FromMinutes(5),
    StaleFor = TimeSpan.FromHours(1),
    Tags = ["customer:7"],
};
Invoice cached = await cache.GetOrCreateJsonAsync(
    "invoice-model:42", token => LoadInvoiceAsync(token), AppJson.Default.Invoice, resilient);
await cache.RemoveByTagAsync("customer:7");   // every entry tagged customer:7 is gone

ClientsRust C#Go Java Python Node.js

PackagesRuvio.Extensions.Caching NuGetRuvio.AspNetCore.OutputCaching NuGet

The 5000 is how long a filler may take. If it crashes halfway, its claim lapses after five seconds and the next caller becomes the filler. The token makes sure that only the caller who was asked to fill can write the value; a late FILLSET with an old token answers 0 and changes nothing.

Serving stale while one caller refreshes

The SWR 30000 on FILLSET is the part I like most. After the value’s 60 seconds are up it is not thrown away at once. For another 30 seconds FILLGET hands out the old copy, and exactly one caller also gets a token to refresh it:

redis-cli

FILLGET product:42 5000
# 1) STALE  2) "{...}"  3) 2     -> stale copy, and you refresh it
FILLGET product:42 5000
# 1) STALE  2) "{...}"  3) (nil) -> stale copy, someone else is refreshing

So the page still renders instantly for everyone, and the database sees one query instead of a burst. If the refresh fails, users keep getting the stale copy until the window ends, which is usually what you want when the database is the thing having a bad day.

Waiting without a retry loop

Without a stale copy, a BUSY caller has nothing to show. It can sleep and try again, or let the server do the waiting:

redis-cli

FILLGET product:42 5000 WAIT 200
# waits up to 200 ms for the filler's FILLSET, then answers HIT or BUSY

From application code

The clients wrap the whole dance in one call: the tabs under the first example show it in every language. In C#, GetOrCreateAsync runs your factory in exactly one caller and polls with short FILLGET calls in the others, so a shared connection is never held.

If you are in ASP.NET Core, the Ruvio.Extensions.Caching package gives you an IDistributedCache with the same GetOrCreate across every instance of your app, plus the stale period and tags you can remove in one call. The details are on the integrations page.

The fine print

Markers in memoryAfter a restart nobody is filling, which is fine: the next caller simply becomes the filler.
Size fill_msA little above your slowest normal load. Too short and two callers may fill; too long and a crashed filler holds everyone up.
The value is a writeFILLSET is an ordinary write with a TTL, as durable as your durability setting makes it.

The animation at the top shows the whole thing: one request goes to the database, the rest are served from memory. The reference for both commands is in the command reference, and the use cases page has the same example in all six clients.