Release 5.17.0

Release 5.17.0 huggingface / transformers Public Notifications You must be signed in to change notification settings Fork 34.7k Star 167k Release 5.17.0 Latest Latest vasqu released this 09 Sep 15:42 · 193 commits to main since this release v5.17.0 856157a Release v5.17.0 New Model additions HYV4 Hy4-Preview is a 780B-parameter mixture-of-experts language model that activates 49B parameters per token. Each MoE layer holds 256 routed experts plus one always-active shared expert and routes every token to 8 of them.

This HardwareLaunch is relevant to the technology intelligence record because it involves GitHub, DeepSeek, NVIDIA, AMD. The source article should remain the factual reference for follow-up coverage.

• huggingface / transformers Public Notifications You must be signed in to change notification settings Fork 34.7k Star 167k Release 5.17.0 Latest Latest vasqu released this 09 Sep 15:42 · 193 commits to main since this release v5.17.0 856157a Release v5.17.0 New Model additions HYV4 Hy4-Preview is a 780B-parameter mixture-of-experts language model that activates 49B parameters per token.
• Each MoE layer holds 256 routed experts plus one always-active shared expert and routes every token to 8 of them.
• The context window is 1M tokens.
• The architecture combines four features: Multi-head Latent Attention (MLA) compresses keys and values into a low-rank latent ( kv_lora_rank ) that kv_b_proj expands back to one key/value per query head.
• DeepSeek Sparse Attention (DSA) selects index_topk keys per query with a lightweight indexer.
• Following IndexShare , only the layers marked "full" in indexer_types run an indexer; "shared" layers reuse the previous full layer's selection.