MoE-Routing-Collapse-Risk-Dataset-Open-Sparse-LLMs

This dataset simulates the self-reinforcing load-imbalance dynamics of Mixture-of-Experts token routers during training. The simulation sweeps six architecture archetypes approximating published open-weight and research MoE configs, Switch Transformer (2022), GShard (2021), GLaM (2022), Mixtral 8x7B (2024), DeepSeekMoE (2024), and OLMoE (2024), across load-balancing-loss weight, expert-capacity factor, and random seed, tracking routing entropy, max-expert token share, and capacity-factor token-drop rate at every one of 200 simulated training steps per run. The underlying router-dynamics equations are grounded in ten papers spanning 2017 to 2024: the original sparsely-gated MoE paper (2017), GShard and V-MoE (2021), Switch Transformers, GLaM, and ST-MoE (2022), Mixtral, DeepSeekMoE, Soft MoE, and OLMoE (2024). All ten sources and every equation are documented in moe_routing_theory.py and the README Research basis. The simulated model architectures reflect 2024-2025 open-weight releases; the mechanisms grounding the equations date back to 2017 since the 2024 models build directly on that earlier work.