A lock is not enough: fencing tokens with LEASE

The usual distributed lock is SET key me NX PX 30000. It works well right up to the day a worker stops for 31 seconds. A long garbage-collection pause, a VM that got migrated, a laptop lid closed during a demo. The lock expires, a second worker takes it, and then the first one wakes up and carries on writing as if nothing happened. For a while you have two holders, and the store that the lock was protecting has no idea.

The fix has been known for a long time: give every grant a number that only goes up, send that number along with every write, and let the thing you protect reject a number older than the last one it saw. That number is a fencing token. In Ruvio every lock grant returns one.

64-bit: fencing token on every grant, and it only grows
3: commands: LEASE, RENEW, RELEASE
0: the answer to RENEW or RELEASE with a token that no longer holds the lock

The three commands

LEASE job:nightly-report 30000
# 114770882368960 -> you hold it, this is your token
LEASE job:nightly-report 30000
# (nil)           -> someone else holds it
RENEW job:nightly-report 114770882368960 30000
# 1               -> deadline moved to now + 30 s
RELEASE job:nightly-report 114770882368960
# 1               -> released

use ruvio_client::Client;

let mut db = Client::connect("127.0.0.1", 6379)?;
let token = db.lease("job:nightly-report", 30_000)?;
// Some(token) -> you hold it
// None     -> someone else holds it
if let Some(token) = token {
    let released = db.release("job:nightly-report", token as u64)?;
    // 1 -> released (0 if the token no longer matches)
}

await using var connection = await RuvioConnection.ConnectAsync("127.0.0.1", 6379);
var db = await connection.GetDefaultDatabaseAsync();

long? token = await db.LeaseAsync("job:nightly-report", 30_000);
// token -> you hold it
// null -> someone else holds it
if (token is long held)
{
    long released = await db.ReleaseAsync("job:nightly-report", held);
    // 1 -> released (0 if the token no longer matches)
}

c, err := ruvio.Connect(ctx, ruvio.Options{Addr: "127.0.0.1:6379"})
if err != nil { return err }

token, ok, err := c.Lease(ctx, "job:nightly-report", 30*time.Second)
if err != nil { return err }
if !ok { return nil } // someone else holds it
// you hold it
released, err := c.Release(ctx, "job:nightly-report", token)
if err != nil { return err }
fmt.Println(released) // true -> released (false if the token no longer matches)

var c = RuvioClient.connect("127.0.0.1", 6379);

Long token = c.lease("job:nightly-report", 30_000);
// token -> you hold it
// null -> someone else holds it
if (token != null) {
    Boolean released = c.release("job:nightly-report", token);
    // true -> released (false if the token no longer matches)
}

import ruvio

r = ruvio.Ruvio("127.0.0.1", 6379)
token = r.lease("job:nightly-report", 30_000)
# token -> you hold it
# None -> someone else holds it
if token is not None:
    released = r.release("job:nightly-report", token)
    # True -> released (False if the token no longer matches)

import { createClient } from 'ruvio-client';

const c = createClient({ host: '127.0.0.1', port: 6379 });
const token = await c.lease('job:nightly-report', 30_000);
// token -> you hold it
// null -> someone else holds it
if (token !== null) {
  const released = await c.release('job:nightly-report', token);
  // true -> released (false if the token no longer matches)
}

services.AddRuvioDistributedLock(options => options.KeyPrefix = "myapp:lock:");

await using IRuvioLock? held = await locks.TryAcquireAsync(
    "nightly-report", lease: TimeSpan.FromSeconds(30), wait: TimeSpan.FromSeconds(5));

if (held is null) return;               // someone else runs it

await RunReportAsync(held.LockLost);    // stops if the lock is lost mid-run

// Or keep a background worker running on exactly one instance; another takes over if it stops.
services.AddRuvioSingleton<OutboxPublisher>("outbox-publisher");

// Or guard an endpoint: one request per order at a time, across every instance.
app.UseRuvioLock();
app.MapPost("/orders/{id}/pay", PayAsync).WithMetadata(new RuvioLockAttribute("order:{id}"));

ClientsRust C#Go Java Python Node.js

PackagesRuvio.AspNetCore.DistributedLock NuGet

LEASE gives you the lock and a token, or a null when someone else has it. RENEW moves the deadline for a long job, but only for the token that holds the lock right now. RELEASE frees it, again only for the right token. A stale worker calling RENEW or RELEASE gets 0 back and changes nothing, so it cannot free a lock that by now belongs to someone else.

Where the token earns its keep

The lock on its own still cannot stop the paused worker. What stops it is the other end. Say the job writes a report row. Keep the highest token you have seen next to the data, and make the write conditional on it:

UPDATE reports
SET body = $1, fence = $2
WHERE id = $3 AND fence < $2;
-- 0 rows updated -> a newer holder already wrote; stop here
</code></pre>
<p>Now follow the animation at the top again. Worker 1 gets token 41 and stalls. The lease runs out, worker 2 gets 42 and writes with it. Worker 1 wakes up and tries to write with 41. The <code>WHERE fence < 41</code> no longer matches, zero rows change, and worker 1 learns that it lost the lock instead of overwriting newer data.</p>
<p>The same trick works with anything that can compare a number: a column in Postgres, an <code>IFGT</code> write in Ruvio itself, a version header on an object store, a check in your own service. The important part is that the comparison happens where the data lives, not in the worker that might be asleep.</p>
<h2 id="when-you-need-n-holders-not-one">When you need N holders, not one</h2>
<p>Sometimes the rule is “at most eight exports at a time” rather than “one”. <code>SEMAPHORE</code> gives each holder its own slot, its own TTL and its own token:</p>
<pre><code class="language-redis">SEMAPHORE upstream:billing 8 15000
# 114770882371136 -> a slot, this is your token
# (nil)           -> all 8 slots are taken
RELEASE upstream:billing 114770882371136

use ruvio_client::Client;

let mut db = Client::connect("127.0.0.1", 6379)?;
let token = db.semaphore("upstream:billing", 8, 15_000)?;
// Some(1700000000123) -> a slot, this is your token
// None                -> all 8 slots are taken
if let Some(token) = token {
    db.release("upstream:billing", token as u64)?;
}

await using var connection = await RuvioConnection.ConnectAsync("127.0.0.1", 6379);
var db = await connection.GetDefaultDatabaseAsync();

long? token = await db.AcquireSemaphoreAsync("upstream:billing", 8, 15_000);
// 1700000000123 -> a slot, this is your token
// null          -> all 8 slots are taken
if (token is long held)
{
    await db.ReleaseAsync("upstream:billing", held);
}

c, err := ruvio.Connect(ctx, ruvio.Options{Addr: "127.0.0.1:6379"})
if err != nil { return err }

token, ok, err := c.Semaphore(ctx, "upstream:billing", 8, 15*time.Second)
if err != nil { return err }
if !ok { return nil } // all 8 slots are taken
// a slot, token is 1700000000123
if _, err := c.Release(ctx, "upstream:billing", token); err != nil { return err }

var c = RuvioClient.connect("127.0.0.1", 6379);

Long token = c.semaphore("upstream:billing", 8, 15_000);
// 1700000000123 -> a slot, this is your token
// null          -> all 8 slots are taken
if (token != null) {
    c.release("upstream:billing", token);
}

import ruvio

r = ruvio.Ruvio("127.0.0.1", 6379)
token = r.semaphore("upstream:billing", 8, 15_000)
# 1700000000123 -> a slot, this is your token
# None          -> all 8 slots are taken
if token is not None:
    r.release("upstream:billing", token)

import { createClient } from 'ruvio-client';

const c = createClient({ host: '127.0.0.1', port: 6379 });
const token = await c.semaphore('upstream:billing', 8, 15_000);
// '1700000000123' -> a slot, this is your token (a decimal string)
// null            -> all 8 slots are taken
if (token !== null) {
  await c.release('upstream:billing', token);
}

ClientsRust C#Go Java Python Node.js

A crashed worker frees its slot when its own TTL ends. The others are not affected, and the key goes away when the last holder leaves.

Failover, honestly

Replication in Ruvio is asynchronous by default. If a primary hands out a lease and dies before the replica has that write, the new primary has never heard of the grant and can hand the same lock to someone else. Two things close that gap:

Take the lock on a connection that sent CLIENT DURABILITY REPLICAS 1. The grant is then only answered once a replica holds it on disk, and ruvio coordinator fails over only to a replica that holds every such write.
Keep the protected side checking tokens. Tokens keep growing across a failover, so the check above still rejects the older holder even in the rare case where two grants overlap.

Keep the nodes’ clocks synchronized as well, since a lease is a deadline. None of this makes a lock magically perfect. It moves the guarantee to the place that can actually enforce it, and that is the whole point of fencing.

Rules of thumb

Size the TTLLonger than a normal run. Long jobs call RENEW with their token instead of guessing a huge TTL.
Release earlyFree the lock as soon as you are done. Waiting for the TTL just makes the next worker wait.
Always fencePass the token to the write you protect. A lock without a fence is a hint, not a guarantee.

The full syntax is in the command reference. If you have a lock story of your own, good or bad, I would like to hear it in Discussions.