Scale from 1 to 3 to 10 nodes

Choose the topology

These private addresses are examples; provision your own reachable endpoints. Use one independent data directory per server and a unique stable shard ID per endpoint. IDs belong to the client route configuration, not to a server --shard-id flag. There is no --join, seed discovery, quorum or replication factor. Full CacheIdentity determines the rendezvous-hash owner; a key is not copied to all nodes.

First, reproduce the local example

Build from the source revision linked below, after the quick-start prerequisites. The example prepares isolated temporary stores, stops and joins the old servers at each transition, advances all epochs, replaces the full client route set, rejects an old session and checks Lookup / Put / Commit / Get / Drained release on every shard. It cleans up its temporary directories.

cargo run -p gpucache-transport --example scaling_smoke
cargo test -p gpucache-transport --example scaling_smoke

Selected lines from a successful run are shown below. The full output also checks every shard: IDs 0, 0–2, and 0–9, respectively. These are correctness results, not performance measurements.

status=SCALING_SMOKE_OK
topology=SINGLE_PROCESS_LOCALHOST
stage_shards=1 store_epoch=1 local_revision=1
stage_shards=3 store_epoch=2 local_revision=2
stage_shards=10 store_epoch=3 local_revision=3
per_shard_roundtrips=14
stale_sessions=REJECTED
automatic_payload_migration=NONE
durability_replication=NONE
hardware=NOT_RUN_HARDWARE
performance=NOT_A_PERFORMANCE_RESULT

Start one node

This loopback command starts an independent FileStore server. The fingerprint below is a public development fixture; use your own credential and tenant policy outside a local test. Configure ShardedClient with shard ID 0, this endpoint and the epoch obtained from the TCP handshake. Use its lease/session API so Put and Commit stay with the selected owner.

cargo run -p gpucache-transport --bin gpucache-tcp -- serve \
  --data-dir ./data/shard-0 --bind 127.0.0.1:9735 \
  --tenant 1 --trust 1 \
  --fingerprint 0000000000000000000000000000000000000000000000000000000000000001

For a remote node, bind its private interface (for example --bind 10.0.0.11:9735 --allow-remote) only on an isolated trusted fabric. --allow-remote enables neither TLS nor peer authentication. The fingerprint is a bearer-equivalent secret over plaintext TCP. The CLI starts a server; application code must construct the sharded client.

Configure the client before the first stage

In application code that already supplies the authenticated context and verified initial endpoints, choose limits for the final topology when constructing the client. ClientLimits cannot be changed by replace_routes. This example configuration allows ten shards and one connection per shard throughout all three stages:

use gpucache_transport::{ClientLimits, ShardedClient};

let limits = ClientLimits {
    max_shards: 10,
    max_connections: 10,
    max_connections_per_shard: 1,
    max_frame_bytes: 2 * 1024 * 1024,
    ..ClientLimits::default()
};
let client = ShardedClient::new(auth, endpoints, limits)?;

The default per-shard connection cap is 4. Reducing max_connections to 3 while leaving that cap unchanged is invalid. Each server defaults to 16 connections shared across all clients; every client process has its own pool, so plan aggregate usage.

The CLI server uses a 2 MiB frame-body limit, matched above by max_frame_bytes. Frame limits are not negotiated: keep client and server limits aligned. The standalone scaling_smoke example instead uses matching 1 MiB library defaults on both sides.

A new client starts at local revision 1 regardless of the actual store epochs. After the coordinated server transition, read that client’s current revision and call client.replace_routes(revision, endpoints) with the complete snapshot. Revisions are local to each client; they are not store epochs or a shared consensus term.

Move from 1 to 3 nodes

1. Prepare new shard IDs 1 and 2 with independent storage, consistent tenant/trust policy and reachable endpoints. Use the client limits chosen before the first stage; check server, pin, byte and retry budgets as well.
2. Pause every client that can reach the old owners. Stop new Lookup/Get/Put/Commit work, drain transfers and release pins with confirmed completion. Expiry alone is not proof that a transfer is drained. Resolve in-flight writes or cancel/fence them; do not carry their leases across the transition.
3. Stop and join all old server workers, or use an externally verified fence that prevents further reads and writes. replace_routes only changes local client state; it cannot fence a remote server. If this step cannot be proven, do not publish new routes.
4. Prepare every store under exclusive ownership so its next server handshake reports an epoch strictly greater than the previous snapshot maximum. Restart old owners and initialize/reopen new stores as needed. Validate the actual handshakes before making them reachable to clients.
5. Build a full snapshot for IDs 0, 1 and 2 with each actual address and epoch. For every paused client read let revision = client.stats().revision; then call client.replace_routes(revision, endpoints). Handle rejection; never send only newly added entries. Resume only after every required client has accepted the same complete snapshot.
6. After the transition, acquire a fresh session for new logical work. Within a valid epoch, keep a generation’s Commit/Get/Release on its original shard and epoch; do not reroute generation-only follow-ups. Discard fenced work rather than rewriting its epoch. Remapped identities have no automatic migration: recompute on miss/error and explicitly repopulate. Optional cold restore stays off the hot path.

Move from 3 to 10 nodes

Repeat the same pause → drain → stop/fence → epoch preparation → complete snapshot → validation → resume sequence for IDs 0–9. For an initial all-1 then all-2 demonstration the next epoch is 3; in an existing deployment always use the observed previous maximum, not the node count. Use the limits provisioned when the client was constructed; route replacement does not resize them. Plan per-tenant limits, retry-table credit, pin/region credit, frame limits, memory and disk capacity per shard; adding nodes does not remove those bounds.

Clients route directly to the selected owner. ShardDirectory contains bounded metadata, not a central payload cache. Loss of one owner is a clean miss for its keys, not replica failover. Three nodes do not imply a quorum and ten nodes do not establish 10k-GPU readiness. Capacity and key distribution must be measured with your workload; this guide makes no linear throughput claim.

Validate, observe and roll back

• Record the source commit, each stable shard ID/address/actual epoch and the complete snapshot version used by every client. Ensure there are no duplicate IDs or endpoints.
• Test a key owned by every shard: Lookup → Put → Commit → Get, byte comparison and Drained release. Verify wrong-tenant/identity requests cannot hit. A generation is shard-local: keep shard ID and epoch with it.
• After a membership change, verify old sessions/leases are rejected and no pending write commits against the previous owner. Kill a test owner and confirm timeout/miss leads to recompute, never a different-identity hit.
• Inspect client statistics and storage drain/quarantine/fence reports. Track logical descriptor bytes separately from wire bytes, quota failures, retries and connection usage. Reuse the repository CPU regression gate; do not treat it as a real-host qualification.
• If publication or a smoke check fails, keep traffic paused. Repair the snapshot or return to the old node set using another coordinated stop/fence and a newer epoch for every endpoint. Never restore an old epoch/snapshot or revive an old lease. Any lost hot entries can be recomputed.

python3 scripts/ci/vm_regression.py --profile cpu --offline \
  --output artifacts/scaling-cpu

Read the source

The repository runbook and executable example are the authority for the API. There is no turnkey cluster manager, online resharding/data migration, rolling-upgrade availability guarantee or production multi-host qualification in this guide.

SCALING.md · scaling_smoke.rs · OPERATIONS.md · SECURITY.md