RustFS GPUCache · G3.5 In development
Shared KV caching at G3.5.
GPUCache is the G3.5 shared KV hot tier. The implemented data path uses host memory and TCP. RustFS is the colder G4 object tier, outside the hot request path.
Rust · Apache 2.0 · CPU/TCP source available
Problem
When context is evicted, the GPU has to recompute it.
Long context, multi-turn conversation, and agents make the KV Cache grow until GPU memory cannot hold it. The next time evicted KV is needed, it has to be prefilled again: the GPU spends cycles on work it has already done, and the first token waits longer.
Each existing tier has a limit. Host memory is finite. A node-local SSD serves one machine and is hard to share across nodes. Shared storage is built for durability and capacity, and it is not on the inference hot path.
Approach
Reuse compatible blocks. Recompute on a miss.
BlueField-3, GDS and real RDMA are targets in development: NOT_RUN_HARDWARE. The current implementation performs host copies and does not claim zero-copy or CPU bypass.
- 01
The path that runs today
CPU/TCP, memory and ordinary-file backends are available in source. No GPU or DPU is required.
- 02
One node or multiple shards
Bounded routes, connections and epoch fencing support operator-managed shards.
- 03
Failure is recompute
G3.5 adds no erasure coding or durability replicas. Unknown transfer completion stays quarantined until explicit drain or a newer fence permits reuse. RustFS owns cold-data durability. This is a cache contract, not disaster recovery or proof of fleet-scale readiness.
Not a fit
- Data that must be stored durably. G3.5 is a cache. Durable data belongs on RustFS at G4.
- Checkpoint storage for training. That is work for G4.
- Production deployment has not been qualified.
Architecture
CPU/TCP today. Hardware targets remain separate.
GPUCache is the G3.5 shared KV hot tier. The implemented data path uses host memory and TCP. RustFS is the colder G4 object tier, outside the hot request path.
- Host client
- TCP
- G3.5 shardRust
- CPU
- Host memory / files
CPU contract available
- 01
The path that runs today
CPU/TCP, memory and ordinary-file backends are available in source. No GPU or DPU is required.
- 02
Identity before reuse
Only committed blocks with the complete expected identity can hit. A miss means recompute.
- 03
One node or multiple shards
Bounded routes, connections and epoch fencing support operator-managed shards.
- 04
Cold storage stays separate
RustFS is G4, off the hot path. Cold mock contracts are tested; live RustFS is NOT_RUN_HARDWARE.
- 05
The DPU target
DPU/GDS/RDMA remain NOT_RUN_HARDWARE. engine-spike uses fake blocks; aarch64 is CPU evidence only.
BlueField-3, GDS and real RDMA are targets in development: NOT_RUN_HARDWARE. The current implementation performs host copies and does not claim zero-copy or CPU bypass. The path that runs today
Roadmap
Where the work stands.
Status separates available source and reproducible CPU contracts from hardware qualification. This roadmap promises neither a release date nor measured fleet capacity.
| Status | Capability | Boundary |
|---|---|---|
| CPU/TCP available | Identity, lifecycle, MemoryStore/FileStore, TCP ACL and reader admission | Synthetic tests; trusted network boundary, no mTLS. |
| CPU/TCP available | Bounded shard clients, membership updates and epoch fencing | Operator-managed routing; no distributed consensus or fleet-size certification. |
| Mock contract available | G4 manifest v3, bounded sink, refill and retention | Live RustFS: NOT_RUN_HARDWARE. Orphan collection is not automatic. |
| CPU simulation available | DPU messages, fault injection and functional TCP comparison | No device offload, zero-copy, throughput or TCO evidence. |
| Fake-block spike available | EngineBlockContract and engine-spike | Real vLLM export/import: NOT_RUN_HARDWARE; GPU/model run: NOT_RUN_BUDGET. |
| In development | Real RDMA, GDS, BlueField-3 / DOCA and SPDK | NOT_RUN_HARDWARE. Optional features remain unlinked and fail closed. |
GPUCache is built by the RustFS team. The two have a clear split in the storage hierarchy. GPUCache is the G3.5 hot tier, close to the GPU and meant for reuse. RustFS is the G4 capacity tier, an S3-compatible distributed object store written in Rust, responsible for keeping models, datasets, session history, and inference artifacts.
A lost hot tier can be recomputed. Data in the capacity tier must not be lost. Keeping the two jobs separate makes each one simpler.
Quick start In development
Build, test and inspect the CPU contract.
The main branch contains runnable Rust code for G3.5 shared KV caching over CPU/TCP. Start with the software contract tests; no GPU or DPU is required.
Source code is available. There is no installable release or production deployment qualification.
- Install stable Rust with Cargo, Git and Python 3.
- Clone the source and run the workspace tests.
- Use the full quick start for demos and the CPU regression.
git clone https://github.com/rustfs/GPUCache.git
cd GPUCache
cargo test --workspace --all-targets
cargo run -p gpucache-transport --bin share-demoThe full quick start includes the CPU regression and explicit hardware limitations. Build and test from source
Docs
Read the implementation and its boundaries.
Architecture, tiering, status, source commands and scaling guidance distinguish CPU contracts from unqualified hardware targets.
Participate
Build the G3.5 tier together.
GPUCache welcomes people who know Rust systems programming, RDMA/RoCEv2, NVIDIA DOCA and BlueField, ARM64 cross-compilation, and the KV Cache mechanisms in vLLM and TensorRT-LLM.