🤖 AI 资讯

NeMo-DCR: Bit-Exact Delta-Compressed Refit for Scalable Agentic RL at Trillion-Parameter Scale

Authors: Songlin Jiang, Zhiyu Li, Terry Kong, Yu Yao, Youngeun Kwon, Bernard Nguyen, Ashwath Aithal, Mario Di Francesco (NVIDIA)

Abstract

Abstract. Agentic reinforcement learning (RL) disaggregates training from rollout, so each policy update must reach the rollout clusters before the next batch. Transferring a full 1T checkpoint for such weight synchronization (refit) takes 87.5 min between two AWS regions. Measurements of BF16 training show that about 1% of weights change their stored values per step. Recent systems exploit this sparsity but fall short on placement, exactness, or efficiency: they reimplement placement rules, assemble full tensors, rebuild values arithmetically, or use a cross-cluster collective, and none fully recovers from mid-refit failures. We present NeMo-DCR (Delta-Compressed Refit), which sends only changes yet is bit-exact: receivers obtain the same parameter and buffer bits as a dense refit. For placement, fixed affine mappings project changes from training shards into the checkpoint’s canonical coordinates, residual conversion covers the other changes, and the serving runtime’s native loader places all changes in receiver storage. For exactness, compressible XOR masks carry affine changes whose projection and loader preserve stored bits, and overwrites carry the others. Receivers apply both in place, retries overwrite partial writes, and a joint commit binds the policy to the baseline for the next delta. For efficiency, object storage or a relay tree streams payloads during delta construction, without a cross-cluster collective. Even at 3% and 5% change rates, NeMo-DCR refits of 30B–1T models are 12–40× faster than a transport-only full-checkpoint reference. A 1T relay-tree refit at 3% takes 150 s instead of 87.5 min, making refits practical for cross-cluster agentic RL at trillion-parameter scale. Documentation: NeMo RL weight refit guide Code: NeMo RL pull request #2444

1 Introduction

Agentic reinforcement learning (RL) spends over 70% of its wall-clock time in rollout, so rollout is disaggregated from training and runs on serving clusters with suitable hardware and parallel layouts [6, 7]. Before the next rollout batch, a refit must deliver the updated policy to the receivers, the rollout ranks that generate responses with a serving runtime [15]. Disaggregation also lets rollout run on idle serving GPUs [6] or in other datacenters [41, 40], where every refit must cross a wide-area network.

A cross-cluster transfer of a 120B model’s 247.2 GB checkpoint through object storage takes 750 s, and at the 1T scale of models like Kimi K2 [14], such a transfer takes 87.5 min (Figure 1). Yet in the six models of Figure 2, only 0.6–1.2% of training-side source elements change their bfloat16 (BF16) values per step. We call this fraction the element change rate. With only 8 significant bits, BF16 rounds most optimizer updates away. Other reports give element change rates of 1.5–2.0% in Qwen3 RL [26] and average deltas of 1.98% of a 1,024 GiB checkpoint [5]. A full-checkpoint transfer thus spends 98–99% of its bytes on unchanged elements.

Figure 1: NeMo-DCR relay-tree refit at a 3% element change rate versus the transport-only full-checkpoint reference: 22.6 s instead of 750 s at 120B and 150 s instead of 87.5 min at 1T (§§ 8.5 and 8.6).

Figure 2: Element change rate per optimizer step, averaged over each model’s first five steps with the setup of Appendix E: at most 1.2% in every model.

(Footnote: footnotetext: Work conducted during an internship at NVIDIA.)

A refit that sends only the changed 1% faces three challenges: placement, exactness, and efficiency.

First, placement: sparse source changes do not tell receivers where or what to write. A training-shard location does not map directly to a receiver’s storage location: refitting may gather, split, or permute tensors between training and rollout layouts [26]. A change to a shared scale factor can also alter converted values even when other source values stay unchanged [23]. Some systems reimplement the serving runtime’s placement rules [6, 26], and others extract a delta after full-tensor assembly and conversion, which handles both problems but pays for both steps [11].

Second, exactness: a smaller transfer must still deliver the exact updated bits, because rollout–training mismatch is known to destabilize RL [28]. Arithmetic reconstruction can introduce floating-point rounding errors [26], while absolute overwrites avoid rounding [19] but resend unchanged bits within changed values. Failures also threaten exactness: applying updates in place avoids a separate receiver-side baseline copy, but an interrupted refit can leave a mixture of old and updated values that matches neither policy version. Because a delta applies only to the version it was computed from [19], recovery must complete the interrupted refit before rollout resumes and keep receivers aligned with the source baseline that the next refit compares against.

Third, efficiency: delivery must overlap the other refit stages without a cross-cluster collective. The next rollout batch waits for the complete update, so without overlap, delta construction and application can offset the transfer savings. A cross-cluster collective couples receiver membership to the training cluster and introduces timeouts upon failure [22]. Some deployments share weights only through object storage [3], and others have direct links between clusters [40], so delivery must support both.

Existing systems meet these challenges only in part: they either send every weight or lack exactness, native placement, decoupled delivery, or recovery, so none meets all five requirements in Table 1. We present NeMo-DCR, a system for bit-exact delta refit that meets all five: it sends only changes, yet every receiver obtains the same parameter and buffer bits as a dense refit that sends every weight. The key idea is to describe changes in canonical coordinates, the tensor names and indices of the Hugging Face checkpoint format that both training and serving already support. Fixed affine index mappings project changes directly into these coordinates and cover over 96% of the weight bytes in the mixture-of-experts (MoE) checkpoints. The serving runtime’s native loader then maps these coordinates to receiver storage, much as a page table maps virtual to physical addresses.