MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA based Mixture of Experts

1 code implementation • 22 Apr 2024 • Dengchun Li, Yingzi Ma, Naizheng Wang, Zhiyuan Cheng, Lei Duan, Jie Zuo, Cal Yang, Mingjie Tang

Unlike other LoRA based MoE methods, MixLoRA enhances model performance by utilizing independently configurable attention-layer LoRA adapters, supporting the use of LoRA and its variants for the construction of experts, and applying auxiliary load balance loss to address the imbalance problem of the router.

Multi-Task Learning Quantization