NVIDIA Maximizes Inference Efficiency with 'Iterative Puzzle' Framework, Halving Large Model Operational Costs

NVIDIA has unveiled an optimized model to address the significant infrastructure costs associated with deploying and operating large language models (LLMs). Based on a hybrid Mamba-Transformer architecture, the new model maintains performance while boosting server throughput by up to 4.6x and enabling high-performance GPUs to handle long-context requests. The company reduced the parameter count of its existing 120.7B-parameter 'Nemotron-3-Super' model to 75.3B parameters and active parameters to 9.3B, achieving 38% and 27% reductions respectively, while preserving inference, coding, multilingual, and long-context capabilities. The 'Iterative Puzzle' compression technique employs multi-stage gradual compression with knowledge distillation, combined with reinforcement learning, quantization, and multi-token prediction to maximize efficiency without compromising accuracy.