Baseten boosts rollout throughput up to 3.2× with in-place weight updates

Top News

Baseten’s previous RL rollout update path paused serving and kept a full checkpoint copy in host memory. Baseten’s delta weight-sync system applies sparse updates directly to serving GPUs, dropping that host-memory copy.

Here is what changed:

• GLM-5.3 sync time fell from 36 to 15 seconds, while the request pause dropped from 6 to 0.8 seconds.
• Removing the CPU checkpoint copy avoids roughly 770 GB of host-memory use on GLM-5.3 and lets the next delta apply when it arrives.
• Checksum verification rejects corrupted downloads or a wrong parent before GPU writes. A Kimi K3 30 GB delta improved from about 230 to 100 seconds.
• Host-memory KV offloading lifted AgentX cache hits from 2% to 92% at concurrency 48, with output throughput per GPU rising 3.2×.

One catch: sync timings used one replica and trainer-shaped fixture deltas. The AgentX throughput gain disappeared at concurrency 16 because the KV cache fit on the GPU.

Sources: baseten.co