NVIDIA Introduces Reshard Collective to Accelerate RL Weight Refit in Disaggregated Clusters
NVIDIA's open-source NCCL M2N Reshard collective optimizes model parameter transfers during RL post-training, reducing refit latency by up to 5.7x.
During reinforcement learning (RL) post-training, systems frequently run inference and training concurrently on separate GPU clusters to maximize operational efficiency. In this disaggregated setup, policy models continuously generate training trajectories through inference while training GPUs update the model parameters. To keep inference instances synchronized with the latest updates, the system must regularly refresh model weights on inference GPUs—a process known as weight refit. However, transferring massive parameter updates across disaggregated clusters introduces severe communication bottlenecks as model sizes scale into trillions of parameters.
The challenge of scaling weight refit
Refit operations present two primary technical challenges: layout misalignment and massive data volume. Because training and inference engines prioritize different computational workloads, they use distinct parallelization strategies, combining data, context, tensor, expert, and pipeline parallelism in different configurations. Consequently, parameters are distributed differently across training and inference instances.
Historically, frameworks relied on a gather-and-broadcast pattern to manage layout mismatches. Under this method, training GPUs execute an all-gather operation to reconstruct the entire parameter tensor locally, broadcast the complete tensor to all inference GPUs, and then have each inference GPU slice out its required shard. This approach forces inference GPUs to receive full tensors even when they only require a fraction of the data, creating massive communication redundancy.
As models expand, this redundant payload becomes unsustainable. For example, Kimi K3 (2.8 trillion parameters) and Qwen 3.8 2.4T A95B (2.4 trillion parameters) require roughly 5.6 TB and 4.8 TB of storage respectively in BF16 format. Using gather-and-broadcast, the logical payload reaches double the model weight size—approximately 11.2 TB for Kimi K3 and 9.6 TB for Qwen 3.8 per refit step. With frontier open-weight models requiring around 10 TB of communication per step, traditional refit mechanisms impose severe latency overheads.
Reformulating refit with PyTorch DTensor and NCCL M2N
To reduce transfers to the theoretical minimum, NVIDIA researchers reformulated weight refit as a PyTorch distributed tensor (DTensor) resharding problem. DTensor abstractions describe global tensor shapes alongside device meshes and placement rules. By viewing refit as converting a DTensor from a training layout to an inference layout, the transfer logic can calculate exact intersections between source and destination GPU shards, sending only the overlapping regions directly.
This abstraction powers the Reshard collective, a single collective communication operation for arbitrary DTensor-to-DTensor transfers. NVIDIA released an open-source reference implementation called NCCL M2N Reshard in the NVIDIA/nccl-extensions repository on GitHub.
The implementation optimizes hardware usage by combining network transfers with local NVLink communication. Instead of transmitting duplicate source shards to every destination replica over the network, NCCL M2N Reshard groups destination GPUs by NVLink domain. The network delivers balanced, non-redundant data subsets across the GPUs within an NVLink domain, which then exchange missing portions locally over high-bandwidth NVLink. Network transfers and NVLink exchanges are pipelined concurrently to hide latency.
NVIDIA integrated this mechanism into its open-source NeMo RL framework using a hybrid strategy. Because feed-forward projection weights account for roughly 95% of parameters in modern transformer architectures, NeMo RL routes these dominant weights through the Reshard collective while handling smaller weights (such as QKV or Mamba layers) via standard gather-and-broadcast.
In benchmarks across DGX H100 and DGX GB200 systems, resharding achieved latency speedups between 2.7x and 5.7x compared to conventional gather-and-broadcast methods. On a DGX H100 system running DeepSeek v3 671B (PP16 x EP16 to TP32 x DP8), refit latency fell from 14.9 seconds to 2.6 seconds. On a DGX GB200, the same model refit dropped from 6.1 seconds to 2.3 seconds. For the 2.4-trillion parameter Qwen3.8 model in BF16, a refit-only microbenchmark completed in 4.3 seconds.
What it means for developers
The introduction of the Reshard collective directly addresses one of the primary infrastructure bottlenecks in scaling reinforcement learning for massive language models. By cutting down communication payloads and drastically reducing refit latency during RL post-training, engineers can iterate faster and train large-scale disaggregated setups more cost-effectively.
For AI engineers working on distributed systems, NeMo RL allows turning on reshard transport by setting refit_transport: nccl_reshard. Meanwhile, developers looking to integrate and evaluate top AI models in their own applications can try top AI models cheaply through one API at https://apixoai.online.
As open-weight model sizes continue to reach into trillions of parameters, reducing GPU idle time during weight synchronization becomes essential. The availability of open-source tools like NCCL M2N Reshard establishes a standardized abstraction for distributed tensor resharding across complex cluster topologies.
Source: Accelerating Reinforcement Learning Weight Refit with the Reshard Collective — NVIDIA Technical Blog. Written by the Apixo team from that report.
One key for Claude, GPT, GLM, DeepSeek and more. Pay per token with crypto.
Get your API key
