NVIDIA's Nemotron-3-Nano-30B Achieves Breakthrough Efficiency in AI Inference with Novel 4-bit Quantization

NVIDIA has announced the availability of Nemotron-Nano-3-30B-A3B-NVFP4, a production-ready checkpoint for a 30-billion parameter reasoning model. This new iteration operates using the advanced 4-bit NVFP4 format, achieving accuracy levels closely mirroring its 16-bit BF16 baseline. The model integrates a unique hybrid architecture combining Mamba2 and Transformer Mixture of Experts (MoE) elements with a specialized Quantization Aware Distillation (QAD) technique tailored for NVFP4 deployment. This combination results in an ultra-efficient version of Nemotron-3-Nano, capable of delivering up to four times greater throughput on NVIDIA's latest Blackwell B200 GPUs.

Understanding Nemotron-3-Nano-30B-A3B-NVFP4

This newly released model represents a quantized variant of the original Nemotron-3-Nano-30B-A3B-BF16, developed from the ground up by NVIDIA. It functions as a versatile model for both reasoning and chat applications. Its hybrid Mamba2 Transformer MoE network boasts:

• A total of 30 billion parameters.
• A deep structure comprising 52 layers.
• 23 layers dedicated to Mamba2 and MoE functionalities.
• 6 grouped query attention layers with dual groups.
• Each MoE layer incorporates 128 routed experts and one shared expert.
• Approximately 3.5 billion active parameters per token, with six experts activated for each token.

The model underwent pre-training on an extensive dataset of 25 trillion tokens, employing a Warmup Stable Decay learning rate schedule. Post-training involved a sophisticated three-stage pipeline:

1. Supervised Fine-Tuning (SFT): Utilizing synthetic and curated data for diverse tasks like code generation, mathematical problem-solving, scientific reasoning, tool calling, instruction following, and structured output generation.
2. Reinforcement Learning (RL): Incorporating synchronous GRPO across multi-step tool use, multi-turn conversations, and structured environments, complemented by RLHF using a generative reward model.
3. Post-Training Quantization (PTQ): Converting to NVFP4 with an FP8 KV cache and a selective high-precision layout, followed by the application of QAD.

Specifically, the NVFP4 checkpoint maintains attention and feeding Mamba layers in BF16 precision for stability, while quantizing the remaining layers to NVFP4 and leveraging FP8 for the KV cache.

The Importance of NVFP4 Precision

NVFP4 is NVIDIA's proprietary 4-bit floating-point format, optimized for both training and inference on contemporary NVIDIA GPUs. Its key advantages include:

• A 2 to 3 times increase in arithmetic throughput compared to FP8.
• Approximately 1.8 times less memory consumption for weights and activations.

NVFP4 enhances the MXFP4 format by reducing the block size from 32 to 16 and introducing a two-level scaling mechanism. This dual-level scaling uses E4M3-FP8 scales per block alongside a FP32 scale per tensor. This design allows the quantizer to adapt more effectively to local data statistics, thereby expanding the dynamic range and minimizing quantization errors. While simple post-training quantization often suffices for very large language models, smaller models, particularly those with intricate post-training pipelines, experience significant accuracy degradation without more advanced training-based recovery methods.

Quantization Aware Distillation (QAD) Explained

Traditional Quantization Aware Training (QAT) integrates pseudo-quantization into the forward pass and reuses the original task loss. However, this approach presents challenges for modern LLMs due to the complexity of multi-stage post-training pipelines and the frequent unavailability of original training data for open models. Quantization Aware Distillation (QAD) offers an alternative by altering the objective rather than the entire pipeline.

In QAD, a frozen BF16 model serves as a 'teacher,' guiding an NVFP4 'student' model. Training focuses on minimizing the Kullback-Leibler divergence between their respective output token distributions, circumventing the need to reproduce the original supervised or reinforcement learning objectives. This methodology offers several benefits:

• It facilitates a more precise alignment of the quantized model with its high-precision teacher than QAT.
• It maintains stability even when the teacher model has undergone multiple stages like fine-tuning, reinforcement learning, and model merging, by simply matching the teacher's final behavior.
• It operates effectively with partial, synthetic, or filtered data, as it only requires input text to query both teacher and student, eliminating the need for original labels or reward models.

Performance Benchmarks

Extensive benchmarks on Nemotron-3-Nano-30B, a model heavily reliant on RL, demonstrate the effectiveness of QAD. Tests across datasets like AA-LCR, AIME25, GPQA-D, LiveCodeBench-v5, and SciCode-TQ reveal that basic NVFP4 post-training quantization (PTQ) and even NVFP4-QAT lead to noticeable accuracy drops. In contrast, NVFP4-QAD successfully recovers performance to near BF16 levels, significantly narrowing the accuracy gap across these demanding reasoning and coding benchmarks.

This release underscores NVIDIA's commitment to pushing the boundaries of AI model efficiency and deployment. The combination of the Nemotron-3-Nano-30B architecture, the NVFP4 format, and the innovative QAD methodology represents a significant leap forward in making powerful AI models more accessible and performant for real-world applications.