Async GRPO with LoRA across HF Jobs: a bucket, a proxy, and no NCCL

Async GRPO with LoRA across HF Jobs: a bucket, a proxy, and no NCCL Async GRPO with LoRA across HF Jobs: a bucket, a proxy, and no NCCL Published September 10, 2026 Update on GitHub Upvote 52 Amine Dirhoussi aminediroHF Quentin Gallouédec qgallouedec Kashif Rasul kashif Sergio Paniego sergiopaniego TL;DR AsyncGRPOTrainer can now train a LoRA adapter and sync only that adapter to vLLM (TRL v1.14). A rank-1 adapter is a few megabytes, so it can travel through a Storage Bucket mounted in every Job instead of over NCCL.

This PolicyChange is relevant to the technology intelligence record because it involves GitHub, Hugging Face, OpenAI, DeepSeek. The source article should remain the factual reference for follow-up coverage.

• Async GRPO with LoRA across HF Jobs: a bucket, a proxy, and no NCCL Published September 10, 2026 Update on GitHub Upvote 52 Amine Dirhoussi aminediroHF Quentin Gallouédec qgallouedec Kashif Rasul kashif Sergio Paniego sergiopaniego TL;DR AsyncGRPOTrainer can now train a LoRA adapter and sync only that adapter to vLLM (TRL v1.14).
• A rank-1 adapter is a few megabytes, so it can travel through a Storage Bucket mounted in every Job instead of over NCCL.
• The trainer and the vLLM replicas run as separate Hugging Face Jobs on separate machines.
• A small proxy in front of the replicas adds the auth header, routes each rollout to the replica that already holds its KV prefix, and broadcasts adapter loads to every replica.
• The AsyncGRPO metrics show where the bottleneck sits.
• Five runs take the same recipe from 3 h 27 min to 53 min for 500 steps.