NVIDIA’s ProRL v2 Advances LLM Reinforcement Learning with Extended Training

NVIDIA has introduced ProRL v2, a cutting-edge advancement in reinforcement learning (RL) designed to enhance the capabilities of large language models (LLMs). The innovation, developed by NVIDIA Research, is aimed at testing the effects of prolonged RL training on LLMs, potentially expanding their capabilities beyond conventional limits.

Innovations in ProRL v2

ProRL v2 represents the latest evolution in prolonged reinforcement learning, featuring advanced algorithms and rigorous regularization. The framework is designed to explore whether LLMs can achieve measurable progress through thousands of additional RL steps. Unlike traditional RL techniques, which often suffer from instability, ProRL v2 employs techniques such as chain-of-thought prompting and tree search, allowing models to exploit existing knowledge more effectively.

Core Features and Techniques

ProRL v2 distinguishes itself with several key features:

• Extended training: Over 3,000 RL steps across five domains, achieving new state-of-the-art performance.
• Stability and robustness: Incorporates KL-regularized trust regions and periodic reference policy resets.
• Verifiable rewards: Every reward signal is programmatically determined and checkable.
• Efficiency: Scheduled cosine length penalties ensure concise outputs.

Performance and Discoveries

NVIDIA’s experiments with ProRL v2 have yielded several groundbreaking results:

• State-of-the-art performance: ProRL v2 3K has set a new benchmark for 1.5B reasoning models.
• Sustained improvement: Metrics like Pass@1 and pass@k have shown continuous improvement with extended RL steps.
• Creative solutions: Outputs show reduced n-gram overlap with pretraining data, indicating genuine innovation.
• Boundary breakthroughs: ProRL has demonstrated strong pass rates even in tasks where base models previously failed.

Comprehensive Results

ProRL v2 was evaluated across various benchmarks, including math and code generation, showing significant performance gains. Even with a reduced training context length, the model’s accuracy improved, highlighting the efficiency of ProRL’s approach.

Conclusion

ProRL v2 offers a reproducible foundation for pushing the boundaries of LLM capabilities. It demonstrates that extended RL training can significantly expand a model’s reasoning capabilities, providing a practical training recipe for researchers and practitioners. As NVIDIA continues to refine and improve its models, the findings suggest a promising future for reinforcement learning in AI.

For more information, visit the NVIDIA blog.

Image source: Shutterstock