88Z AI, 3 July — NVIDIA has open-sourced the Nemotron-Labs-TwoTower model, utilizing a 60B parameter dual-tower architecture to address token generation bottlenecks. This discrete diffusion language model combines a 30B autoregressive context tower with a 30B diffusion denoising tower, enhancing efficiency through cross-attention collaboration.
At a glance
- Uses a 60B parameter dual-tower architecture with only 3B parameters activated per tower for efficiency
- Addresses generation bottlenecks through layer-wise cross-attention collaboration between a frozen context tower and a denoising tower
The update
NVIDIA has officially open-sourced Nemotron-Labs-TwoTower, a discrete diffusion language model based on a pretrained autoregressive backbone, designed to break through performance bottlenecks in token generation speed for large models. The model features a total of 60 billion parameters and innovatively adopts a dual-tower architecture. Specifically, it includes a 30-billion-parameter autoregressive context tower and a 30-billion-parameter diffusion denoising tower, with only 3 billion parameters activated in each tower during operation. The context tower remains frozen to maintain autoregressive contextual information, while the denoising tower is dedicated to processing noise block denoising. The two towers collaborate closely through a layer-wise cross-attention mechanism to jointly optimize the generation process.
Sources & editorial note
This article is an AI-assisted synthesis by 88Z AI based on the public sources below. Product claims and availability are attributed to their publishers. It does not represent independent testing or first-hand reporting.
- IT之家Source published
Source dates above refer to the original announcements; the date under the headline is this article’s publication on 88Z AI.