Streaming datasets: 100x More Efficient

Streaming datasets: 100x More Efficient Streaming datasets: 100x More Efficient Published October 27, 2025 Update on GitHub Upvote 87 Andres Marafioti andito Quentin Lhoest lhoestq ben burtenshaw burtenshaw Pedro Cuenca pcuenq merve merve TLDR We boosted load_dataset('dataset', streaming=True) , streaming datasets without downloading them with one line of code! Start training on multi-TB datasets immediately, without complex setups, downloading, no "disk out of space", or 429 “stop requesting!” errors.

This ProductUpdate is relevant to the technology intelligence record because it involves GitHub, Hugging Face. The source article should remain the factual reference for follow-up coverage.

• Streaming datasets: 100x More Efficient Published October 27, 2025 Update on GitHub Upvote 87 Andres Marafioti andito Quentin Lhoest lhoestq ben burtenshaw burtenshaw Pedro Cuenca pcuenq merve merve TLDR We boosted load_dataset('dataset', streaming=True) , streaming datasets without downloading them with one line of code!
• Start training on multi-TB datasets immediately, without complex setups, downloading, no "disk out of space", or 429 “stop requesting!” errors.
• Outrunning our local SSDs when training on 64xH100 with 256 workers downloading data.
• We've improved streaming to have 100x fewer requests, → 10× faster data resolution → 2x sample/sec, → 0 worker crashes at 256 concurrent workers.
• Loading data, especially at the terabyte scale, is a major pain in any machine learning workflow.
• We suffered this while training SmolLM3 , at one point we had to wait 3 hours before each run to download enough data.