Long-run tests

Short tests run in seconds on one machine. These runs keep a multi-node cluster under load for hours, break it on a schedule and then check every recorded operation against what the cluster holds at the end, or measure a whole machine as shards are added.

2026-10-26

Throughput as shards are added

Passed

Where: Azure F16as_v6 (16 cores): servers on 8 cores, load on the other 8
How long: 24 minutes, 3 rounds of every row

Tested

SET and GET at 1, 4 and 8 shards with a cluster-aware client
A client that does not route: every connection on one port, the server forwards each command
One shard: SET, GET, INCR, LPUSH and LPOP

Result

4 shards: 6.8M SET and 5.4M GET per second, against 1.25M and 1.5M on one shard
One port: 4 shards 0.88M SET / 1.05M GET, 8 shards 1.22M SET per second, up from 0.65M after the forwarding fix
One shard: SET, GET and INCR at 1.09M, LPUSH 0.92M, LPOP 0.86M per second at pipeline 16; the full tables are on the benchmarks page

2026-10-26

Memory under a steady load

Passed

Where: Azure, 3 nodes (4 vCPU, 16 GB each) and a load machine
How long: 2 hours

Tested

5.1 million reads and writes on a data set of fixed size, so memory growth cannot come from the data
Faults while the load runs: network delay and packet loss, replicas killed and restarted
Process memory and the server's own count of its data, sampled every 10 seconds

Result

Writes that waited for both replicas: none lost, duplicated or out of order
Every fault recovered, the slowest in 30 seconds
The server's count of its data stayed flat; each process stayed under 17 MB, growing at most 0.6 MB an hour

2026-10-26

Failures under load

Passed

Where: Azure, 3 nodes (4 vCPU, 16 GB each) and a load machine
How long: 3 hours

Tested

9.1 million counter, list and register operations, every one recorded and checked at the end
23 faults of seven kinds: primary, replica and coordinator leader killed, a whole machine killed, a replica frozen, a machine cut off the network, network delay and packet loss

Result

Writes that waited for both replicas: none lost, duplicated, invented or out of order
Every fault recovered, the slowest in 33 seconds
Without the replica wait, a failover can lose the last acknowledged writes, as documented for that mode

How a run works

Each run builds the server from source on fresh virtual machines, starts a coordinator and two replicated groups on three machines, and drives load from a fourth. Every operation is written to a history file with its outcome: acknowledged, refused, or unknown. Faults are injected and healed on a schedule, and after each one the cluster must agree on its primaries and take writes again. At the end every key is read back and the history is checked for lost, duplicated, invented and out-of-order writes and for stale reads.

The tool, its plans and the machine templates are in tests/soak; anyone can run them.