Docs
Tiering
The names below are the usual ones for inference KV Cache tiering. GPUCache is G3.5. It is not another name for G4.
Current implementation: CPU/TCP with host copies; persistent FileStore is a CPU reference backend. The DPU/NVMe/GDS layout in the table is a design target. Real RDMA, GDS and BlueField-3 remain NOT_RUN_HARDWARE. Cold tests use a mock; live RustFS integration is unqualified.
| Tier | Medium | Scope | Role for KV Cache | Product |
|---|---|---|---|---|
| G1 | GPU HBM | One GPU | Hot KV used while generating | — |
| G2 | Host DRAM | One node | Staging and buffer for KV unloaded from HBM | — |
| G3 | Node-local NVMe SSD | One node | Warm KV for short-term reuse. Not shared across nodes. | — |
| G3.5 | Shared NVMe flash managed by a DPU and reached over Ethernet or RDMA | Shared inside a GPU pod or cluster | A hot tier of reusable KV Cache, brought back to G2 or G1 when needed. A loss is a miss; recompute, then write back. | GPUCache (in development) |
| G4 | Shared network storage (object storage) | Data-center scale | Durable cold data: models, datasets, session history, inference artifacts | RustFS |
The same stack, drawn out
- G1HBM
- G2DRAM
- G3NVMe
- G3.5GPUCache
- G4RustFS
G3.5 is the Signal block. It is easier to share than the node-local SSD at G3, and closer to the inference hot path than the shared store at G4.
The boundary between G3.5 and G4
GPUCache does not take on durability. When a G3.5 node fails or data is lost, the inference framework sees a miss, prefills again the way it already would, writes the result back to GPUCache, and the next use can hit.
Context and artifacts that have to be kept for a long time go to RustFS at G4. The hot tier can be recomputed. The capacity tier is where data must remain.