Mixture of Experts

ICLR 2024arXiv:2308.00951

#2

From Sparse to Soft Mixtures of Experts

Joan Puigcerver, Carlos Riquelme Ruiz, Basil Mustafa et al.

208

ICLR 2025arXiv:2409.16040

#3

Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Experts

Xiaoming Shi, Shiyu Wang, Yuqi Nie et al.

mixture of expertstime series forecastingfoundation modelsdecoder-only transformer+4

194

CVPR 2024arXiv:2403.11549

#4

Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapters

Jiazuo Yu, Yunzhi Zhuge, Lu Zhang et al.

190

ICLR 2025arXiv:2409.02060

#5

OLMoE: Open Mixture-of-Experts Language Models

Niklas Muennighoff, Luca Soldaini, Dirk Groeneveld et al.

169

ICML 2024arXiv:2402.01739

#6

OpenMoE: An Early Effort on Open Mixture-of-Experts Language Models

Fuzhao Xue, Zian Zheng, Yao Fu et al.

mixture-of-experts modelslarge language modelsrouting mechanismscost-effectiveness trade-off+3

160

ICLR 2024arXiv:2309.05444

#7

Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuning

Ted Zadouri, Ahmet Üstün, Arash Ahmadian et al.

143

ICLR 2024arXiv:2404.13628

#8

Mixture of LoRA Experts

xun wu, Shaohan Huang, Furu Wei

136

ICML 2024arXiv:2402.07871

#9

Scaling Laws for Fine-Grained Mixture of Experts

Jan Ludziejewski, Jakub Krajewski, Kamil Adamczewski et al.

mixture of expertsscaling lawscomputational efficiencymodel granularity+2

120

ICLR 2025arXiv:2408.15998

#10

Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoders

Min Shi, Fuxiao Liu, Shihao Wang et al.

multimodal large language modelsvision encodersoptical character recognitiondocument analysis+4

116

NEURIPS 2025arXiv:2502.13189

#11

MoBA: Mixture of Block Attention for Long-Context LLMs

Enzhe Lu, Zhejun Jiang, Jingyuan Liu et al.

attention mechanismlong-context llmsmixture of expertssparse attention+2

109

ICML 2024arXiv:2402.00433

#12

Merging Multi-Task Models via Weight-Ensembling Mixture of Experts

Anke Tang, Li Shen, Yong Luo et al.

model mergingtask arithmeticmixture of expertsparameter interference+3

84

ICLR 2025arXiv:2306.03745

#13

Soft Merging of Experts with Adaptive Routing

Haokun Liu, Muqeeth Mohammed, Colin Raffel

conditional computationsparsely activated networksexpert routinggradient estimation+3

82

CVPR 2024arXiv:2404.09502

#14

SparseOcc: Rethinking Sparse Latent Representation for Vision-Based Semantic Occupancy Prediction

Pin Tang, Zhongdao Wang, Guoqing Wang et al.

81

ICLR 2024arXiv:2310.01334

#15

Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policy

Pingzhi Li, Zhenyu Zhang, Prateek Yadav et al.

77

ICML 2025arXiv:2410.10469

#16

Moirai-MoE: Empowering Time Series Foundation Models with Sparse Mixture of Experts

Xu Liu, Juncheng Liu, Gerald Woo et al.

71

CVPR 2024arXiv:2403.12494

#17

Task-Customized Mixture of Adapters for General Image Fusion

Pengfei Zhu, Yang Sun, Bing Cao et al.

69

ICML 2024arXiv:2402.08609

#18

Mixtures of Experts Unlock Parameter Scaling for Deep RL

Johan Obando Ceron, Ghada Sokar, Timon Willi et al.

mixture of expertsparameter scalingreinforcement learningvalue-based networks+2

64

CVPR 2024arXiv:2403.17749

#19

Multi-Task Dense Prediction via Mixture of Low-Rank Experts

Yuqi Yang, Peng-Tao Jiang, Qibin Hou et al.

multi-task dense predictionmixture of expertslow-rank adaptationparameter sharing+3

60

ICML 2024arXiv:2405.01029

#20

MVMoE: Multi-Task Vehicle Routing Solver with Mixture-of-Experts

Jianan Zhou, Zhiguang Cao, Yaoxin Wu et al.

vehicle routing problemsmixture-of-expertsmulti-task learningzero-shot generalization+2

59

ICML 2025arXiv:2503.17547

#21

Learning Multi-Level Features with Matryoshka Sparse Autoencoders

Bart Bussmann, Noa Nabeshima, Adam Karvonen et al.

58

ICLR 2025arXiv:2402.07033

#22

Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Models

Keisuke Kamahori, Tian Tang, Yile Gu et al.

mixture-of-experts modelscpu-gpu orchestrationresource-efficient inferencelarge language models+3

48

ICLR 2024arXiv:2401.10191

#23

Divide and not forget: Ensemble of selectively trained experts in Continual Learning

Grzegorz Rypeść, Sebastian Cygert, Valeriya Khan et al.

46

ICLR 2025arXiv:2406.16437

#24

Theory on Mixture-of-Experts in Continual Learning

Hongbo Li, Sen Lin, Lingjie Duan et al.

continual learningcatastrophic forgettingmixture-of-expertsoverparameterized linear regression+3

44

ICLR 2024arXiv:2310.01542

#25

Fusing Models with Complementary Expertise

Hongyi Wang, Felipe Polo, Yuekai Sun et al.

41

ICLR 2024arXiv:2311.11385

#26

Multi-Task Reinforcement Learning with Mixture of Orthogonal Experts

Ahmed Hendawy, Jan Peters, Carlo D'Eramo

41

ICML 2025arXiv:2410.11842

#27

MoH: Multi-Head Attention as Mixture-of-Head Attention

Peng Jin, Bo Zhu, Li Yuan et al.

40

AAAI 2024arXiv:2306.04990

#28

Multi-Architecture Multi-Expert Diffusion Models

Yunsung Lee, Jin-Young Kim, Hyojun Go et al.

diffusion modelsmulti-expert modelsattention mechanismimage generation+3

38

ICLR 2025arXiv:2408.15881

#29

LLaVA-MoD: Making LLaVA Tiny via MoE-Knowledge Distillation

Fangxun Shu, Yue Liao, Lei Zhang et al.

knowledge distillationmixture of expertsmultimodal language modelspreference optimization+3

38

ICLR 2024arXiv:2403.02600

#30

TESTAM: A Time-Enhanced Spatio-Temporal Attention Model with Mixture of Experts

Hyunwook Lee, Sungahn Ko

38

NEURIPS 2025arXiv:2402.15751

#31

Sparse MeZO: Less Parameters for Better Performance in Zeroth-Order LLM Fine-Tuning

Yong Liu, Zirui Zhu, Chaoyu Gong et al.

zeroth-order optimizationmemory-efficient fine-tuninglarge language modelsparameter-efficient fine-tuning+3

37

COLM 2025arXiv:2406.14909

#32

Mixture of Attention Spans: Optimizing LLM Inference Efficiency with Heterogeneous Sliding-Window Lengths

Tianyu Fu, Haofeng Huang, Xuefei Ning et al.

Efficient AttentionSparse AttentionKV Cache ManagementLarge Language Models+1

37

ICLR 2025arXiv:2405.14297

#33

Dynamic Mixture of Experts: An Auto-Tuning Approach for Efficient Transformer Models

Yongxin Guo, Zhenglin Cheng, Xiaoying Tang et al.

sparse mixture of expertstransformer efficiencyadaptive gating mechanismvision-language tasks+1

36

ICLR 2025arXiv:2410.07348

#34

MoE++: Accelerating Mixture-of-Experts Methods with Zero-Computation Experts

Peng Jin, Bo Zhu, Yuan Li et al.

mixture-of-experts methodszero-computation expertsgating residualsexpert load imbalance+2

36

ECCV 2024arXiv:2403.07508

#35

MoAI: Mixture of All Intelligence for Large Language and Vision Models

Byung-Kwan Lee, Beomchan Park, Chae Won Kim et al.

instruction-tuned llvmsvisual perception tasksscene graph generationoptical character recognition+4

34

CVPR 2024arXiv:2312.00968

#36

Omni-SMoLA: Boosting Generalist Multimodal Models with Soft Mixture of Low-rank Experts

Jialin Wu, Xia Hu, Yaqing Wang et al.

34

CVPR 2025arXiv:2411.18466

#37

Complexity Experts are Task-Discriminative Learners for Any Image Restoration

Eduard Zamfir, Zongwei Wu, Nancy Mehta et al.

image restorationmixture-of-expertscomputational complexitytask-specific allocation+4

32

ICLR 2025arXiv:2412.14711

#38

ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routing

Ziteng Wang, Jun Zhu, Jianfei Chen

mixture-of-experts modelsdifferentiable routingsparse activationdynamic computation allocation+4

31

ICML 2024arXiv:2402.03412

#39

See More Details: Efficient Image Super-Resolution by Experts Mining

Eduard Zamfir, Zongwei Wu, Nancy Mehta et al.

image super-resolutionexpert mininglow-rank expertsrank-wise features+3

30

ICLR 2025arXiv:2410.08201

#40

Efficient Dictionary Learning with Switch Sparse Autoencoders

Anish Mudide, Josh Engels, Eric Michaud et al.

30

ICLR 2025arXiv:2412.12953

#41

Efficient Diffusion Transformer Policies with Mixture of Expert Denoisers for Multitask Learning

Moritz Reuss, Jyothish Pari, Pulkit Agrawal et al.

30

ICLR 2024arXiv:2402.13412

#42

Scaling physics-informed hard constraints with mixture-of-experts

Nithin Chalapathi, Yiheng Du, Aditi Krishnapriyan

ICML 2025arXiv:2501.12370

#43

Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Models

Samira Abnar, Harshay Shah, Dan Busbridge et al.

ICLR 2025arXiv:2409.03277

#44

ChartMoE: Mixture of Diversely Aligned Expert Connector for Chart Understanding

Zhengzhuo Xu, Bowen Qu, Yiyan Qi et al.

ICLR 2025arXiv:2410.10814

#45

Your Mixture-of-Experts LLM Is Secretly an Embedding Model for Free

Ziyue Li, Tianyi Zhou

mixture-of-experts llmsembedding modelsexpert routersrouting weights+4

AAAI 2025arXiv:2412.10650

#46

DeMo: Decoupled Feature-Based Mixture of Experts for Multi-Modal Object Re-Identification

Yuhao Wang, Yang Liu, Aihua Zheng et al.

27

ICCV 2025arXiv:2504.07951

#47

Scaling Laws for Native Multimodal Models

Mustafa Shukor, Enrico Fini, Victor Guilherme Turrisi da Costa et al.

ICML 2024arXiv:2310.14188

#48

A General Theory for Softmax Gating Multinomial Logistic Mixture of Experts

Huy Nguyen, Pedram Akbarian, TrungTin Nguyen et al.

mixture-of-experts modelsoftmax gatingmultinomial logistic regressiondensity estimation convergence+4

ICML 2025arXiv:2501.01230

#49

Modeling Multi-Task Model Merging as Adaptive Projective Gradient Descent

Yongxian Wei, Anke Tang, Li Shen et al.

ICLR 2024arXiv:2309.13850

#50

Statistical Perspective of Top-K Sparse Softmax Gating Mixture of Experts

Huy Nguyen, Pedram Akbarian Saravi, Fanqi Yan et al.

CVPR 2024arXiv:2404.16030

#51

MoDE: CLIP Data Experts via Clustering

Jiawei Ma, Po-Yao Huang, Saining Xie et al.

contrastive language-image pretrainingdata clusteringmixture of expertsnoisy web data+3

25

ICLR 2025arXiv:2410.06270

#52

Efficient Deweahter Mixture-of-Experts with Uncertainty-Aware Feature-Wise Linear Modulation

Rongyu Zhang, Yulin Luo, Jiaming Liu et al.

Mixture Compressor for Mixture-of-Experts LLMs Gains More

Wei Huang, Yue Liao, Jianhui Liu et al.

mixture-of-experts modelsmodel compressionmixed-precision quantizationdynamic pruning+4

24

CVPR 2024arXiv:2404.00417

#54

Orchestrate Latent Expertise: Advancing Online Continual Learning with Multi-Level Supervision and Reverse Self-Distillation

Hongwei Yan, Liyuan Wang, Kaisheng Ma et al.

23

ICML 2024arXiv:2402.02952

#55

On Least Square Estimation in Softmax Gating Mixture of Experts

Huy Nguyen, Nhat Ho, Alessandro Rinaldo

mixture of expertssoftmax gating functionleast squares estimationstrong identifiability+4

23

NEURIPS 2025arXiv:2505.12082

#56

Model Merging in Pre-training of Large Language Models

Yunshui Li, Yiyuan Ma, Shen Yan et al.

21

CVPR 2025arXiv:2501.01709

#57

MoVE-KD: Knowledge Distillation for VLMs with Mixture of Visual Encoders

jiajun cao, Yuan Zhang, Tao Huang et al.

21

ICML 2025arXiv:2502.17298

#58

Delta Decompression for MoE-based LLMs Compression

Hao Gu, Wei Li, Lujun Li et al.

21

ICML 2024arXiv:2401.13875

#59

Is Temperature Sample Efficient for Softmax Gaussian Mixture of Experts?

Huy Nguyen, Pedram Akbarian, Nhat Ho

mixture of expertsdense-to-sparse gatingmaximum likelihood estimationgaussian mixture models+4

20

AAAI 2024arXiv:2308.11971

#60

EVE: Efficient Vision-Language Pre-training with Masked Prediction and Modality-Aware MoE

Junyi Chen, Longteng Guo, Jia Sun et al.

vision-language pre-trainingmasked signal modelingmultimodal transformermodality-aware moe+3

20

ICLR 2024arXiv:2306.16788

#61

Sparse Model Soups: A Recipe for Improved Pruning via Model Averaging

Max Zimmer, Christoph Spiegel, Sebastian Pokutta

19

ICLR 2025arXiv:2410.21317

#62

MatExpert: Decomposing Materials Discovery By Mimicking Human Experts

Qianggang Ding, Santiago Miret, Bang Liu

19

ICLR 2025arXiv:2406.12034

#63

Self-MoE: Towards Compositional Large Language Models with Self-Specialized Experts

Junmo Kang, Leonid Karlinsky, Hongyin Luo et al.

self-specialized expertsmixture of expertsself-generated synthetic dataself-optimized routing+3

19

ICLR 2025arXiv:2410.02098

#64

EC-DIT: Scaling Diffusion Transformers with Adaptive Expert-Choice Routing

Haotian Sun, Tao Lei, Bowen Zhang et al.

18

ICML 2025arXiv:2410.14972

#65

MENTOR: Mixture-of-Experts Network with Task-Oriented Perturbation for Visual Reinforcement Learning

Suning Huang, Zheyu Zhang, Tianhai Liang et al.

18

ICLR 2024arXiv:2311.02684

#66

Octavius: Mitigating Task Interference in MLLMs via LoRA-MoE

Zeren Chen, ziqin wang, zhen wang et al.

18

ICLR 2025arXiv:2405.14260

#67

Graph Sparsification via Mixture of Graphs

Guibin Zhang, Xiangguo SUN, Yanwei Yue et al.

graph sparsificationgraph neural networksmixture of expertsgrassmann manifold+4

17

AAAI 2025arXiv:2412.11085

#68

GraphMoRE: Mitigating Topological Heterogeneity via Mixture of Riemannian Experts

Zihao Guo, Qingyun Sun, Haonan Yuan et al.

graph representation learningtopological heterogeneitymixed curvature spacesriemannian manifolds+4

17

ICML 2024arXiv:2407.00256

#69

One Prompt is not Enough: Automated Construction of a Mixture-of-Expert Prompts

Ruochen Wang, Sohyun An, Minhao Cheng et al.

instruction design automationmixture-of-expert paradigmin-context learningkernel regression theory+3

16

AAAI 2025arXiv:2408.11396

#70

MoE-LPR: Multilingual Extension of Large Language Models Through Mixture-of-Experts with Language Priors Routing

Hao Zhou, Zhijun Wang, Shujian Huang et al.

15

CVPR 2025arXiv:2309.03904

#71

Exploring Sparse MoE in GANs for Text-conditioned Image Synthesis

Jiapeng Zhu, Ceyuan Yang, Kecheng Zheng et al.

15

CVPR 2025arXiv:2501.04004

#72

LiMoE: Mixture of LiDAR Representation Learners from Automotive Scenes

Xiang Xu, Lingdong Kong, hui shuai et al.

ICLR 2025arXiv:2411.12364

#73

Ultra-Sparse Memory Network

Zihao Huang, Qiyang Min, Hongzhi Huang et al.

AAAI 2025arXiv:2412.12704

#74

MapExpert: Online HD Map Construction with Simple and Efficient Sparse Map Element Expert

Dapeng Zhang, Dayu Chen, Peng Zhi et al.

hd map constructionautonomous driving systemssparse map elementsbird's-eye view features+3

ICML 2025arXiv:2502.05172

#75

Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient

Jan Ludziejewski, Maciej Pióro, Jakub Krajewski et al.

ICLR 2025arXiv:2410.19034

#76

Mixture of Parrots: Experts improve memorization more than reasoning

Samy Jelassi, Clara Mohri, David Brandfonbrener et al.

mixture-of-experts architecturetransformer modelsreasoning capabilitiesmemorization performance+3

anomaly detectionparameter-efficient modelsvision transformersmixture of experts+3

#77

MoEAD: A Parameter-efficient Model for Multi-class Anomaly Detection

Shiyuan Meng, Wenchao Meng, Qihang Zhou et al.

ECCV 2024

NEURIPS 2025arXiv:2502.04510

#78

Heterogeneous Swarms: Jointly Optimizing Model Roles and Weights for Multi-LLM Systems

Shangbin Feng, Zifeng Wang, Palash Goyal et al.

multi-llm systemsmodel role optimizationmodel weight optimizationdirected acyclic graphs+4

13

ICLR 2025arXiv:2410.02475

#79

Efficient Residual Learning with Mixture-of-Experts for Universal Dexterous Grasping

Ziye Huang, Haoqi Yuan, Yuhui Fu et al.

residual policy learningmixture-of-experts frameworkdexterous graspingrobot learning+3

13

ECCV 2024arXiv:2409.15557

#80

Mixture of Efficient Diffusion Experts Through Automatic Interval and Sub-Network Selection

Alireza Ganjdanesh, Yan Kang, Yuchen Liu et al.

diffusion probabilistic modelsdenoising timestep clusteringmixture of expertsmodel pruning+4

AAAI 2025arXiv:2412.19108

#81

Graph Mixture of Experts and Memory-augmented Routers for Multivariate Time Series Anomaly Detection

Xiaoyu Huang, Weidong Chen, Bo Hu et al.

CVPR 2025arXiv:2501.01808

#82

MoEE: Mixture of Emotion Experts for Audio-Driven Portrait Animation

Huaize Liu, WenZhang Sun, Donglin Di et al.

CVPR 2025arXiv:2411.14519

#83

Tra-MoE: Learning Trajectory Prediction Model from Multiple Domains for Adaptive Policy Conditioning

Jiange Yang, Haoyi Zhu, Yating Wang et al.

trajectory predictionmulti-domain learningsparsely-gated moeadaptive policy conditioning+3

ICML 2025arXiv:2503.07639

#84

Mixture of Experts Made Intrinsically Interpretable

Xingyi Yang, Constantin Venhoff, Ashkan Khakzar et al.

ICML 2025arXiv:2502.16894

#85

Make LoRA Great Again: Boosting LoRA with Adaptive Singular Values and Mixture-of-Experts Optimization Alignment

Chenghao Fan, zhenyi lu, Sichen Liu et al.

ICML 2025arXiv:2505.03804

#86

MoEQuant: Enhancing Quantization for Mixture-of-Experts Large Language Models via Expert-Balanced Sampling and Affinity Guidance

Zhixuan Chen, Xing Hu, Dawei Yang et al.

ICLR 2025arXiv:2405.14131

#87

Statistical Advantages of Perturbing Cosine Router in Mixture of Experts

Huy Nguyen, Pedram Akbarian Saravi, Trang Pham et al.

ICML 2025arXiv:2505.05799

#88

MxMoE: Mixed-precision Quantization for MoE with Accuracy and Performance Co-Design

Haojie Duanmu, Xiuhong Li, Zhihang Yuan et al.

ICLR 2025arXiv:2410.10626

#89

Efficiently Democratizing Medical LLMs for 50 Languages via a Mixture of Language Family Experts

Guorui Zheng, Xidong Wang, Juhao Liang et al.

medical large language modelsmultilingual adaptationlow-resource languagesmixture of experts+4

ICCV 2025arXiv:2507.04631

#90

Learning Robust Stereo Matching in the Wild with Selective Mixture-of-Experts

Yun Wang, Longguang Wang, Chenghao Zhang et al.

AAAI 2025arXiv:2410.19225

#91

Hierarchical Mixture of Experts: Generalizable Learning for High-Level Synthesis

Weikai Li, Ding Wang, Zijian Ding et al.

high-level synthesisfield programmable gate arraygraph neural networksmixture of experts+3

mixture of expertsdistributed trainingall-to-all communicationexpert routing+2

#92

EMOE: Modality-Specific Enhanced Dynamic Emotion Experts

Yiyang Fang, Wenke Huang, Guancheng Wan et al.

NetMoE: Accelerating MoE Training through Dynamic Sample Placement

Xinyi Liu, Yujie Wang, Fangcheng Fu et al.

ICLR 2025

NEURIPS 2025arXiv:2505.22323

#94

Advancing Expert Specialization for Better MoE

Hongcan Guo, Haolang Lu, Guoshun Nan et al.

mixture-of-experts modelsexpert specializationrouting mechanismsload balancing loss+3

ICLR 2025arXiv:2405.16869

#95

Multiple Heads are Better than One: Mixture of Modality Knowledge Experts for Entity Representation Learning

Yichi Zhang, Zhuo Chen, Lingbing Guo et al.

multi-modal knowledge graphsentity representation learningknowledge graph completionmodality fusion+3

ICLR 2025arXiv:2402.02933

#96

Mixture of Weak and Strong Experts on Graphs

Hanqing Zeng, Hanjia Lyu, Diyi Hu et al.

Intrinsic User-Centric Interpretability through Global Mixture of Experts

Vinitra Swamy, Syrielle Montariol, Julian Blackwell et al.

interpretable neural networksconditional computationmixture of expertssparse feature activation+4

ICLR 2024arXiv:2311.04640

#98

Object-Centric Learning with Slot Mixture Module

Daniil Kirilenko, Vitaliy Vorobyov, Aleksey Kovalev et al.

ICML 2024arXiv:2405.16646

#99

A Provably Effective Method for Pruning Experts in Fine-tuned Sparse Mixture-of-Experts

Mohammed Nowaz Rabbani Chowdhury, Meng Wang, Kaoutar El Maghraoui et al.

mixture of expertsmodel pruningexpert pruningsparse gating+3

AAAI 2025arXiv:2504.00458

#100

Mixture-of-Attack-Experts with Class Regularization for Unified Physical-Digital Face Attack Detection

Shunxin Chen, Ajian Liu, Junze Zheng et al.