7 accepted papers
Traditional Mixture-of-Experts (MoE) models lack explicit constraints to ensure the router’s decisions align well with the experts’ capabilities, which ultimately limits model performance. To address this, we propose expert-router coupling loss (ERC loss), a lightweight auxiliary loss that couples e…
Existing scaling laws for Large Language Models (LLMs), predominantly monotonic power laws, have successfully guided model development but fail to explain emerging non-monotonic phenomena such as catastrophic overtraining and quantization-induced degradation, where performance deteriorates despite i…
Progressive Learning (PL) reduces pre-training computational overhead by gradually increasing model scale. While prior work has extensively explored depth expansion, width expansion remains significantly understudied, with the few existing methods limited to the early stages of training. However, ex…
Large language models (LLMs) encounter computational challenges during long-sequence inference, especially in the attention pre-filling phase, where the complexity grows quadratically with the prompt length. Previous efforts to mitigate these challenges have relied on fixed sparse attention patterns…
Model merging has emerged as a promising technique for enhancing large language models, though its application in large-scale pre-training remains relatively unexplored. In this paper, we present a comprehensive investigation of model merging techniques during the pre-training process. Through exten…
A novel origami-inspired foldable pneumatic actuator is proposed in this letter to satisfy the comprehensive requirements in wearable assistive application. The pneumatic actuator combines the origami structure of designed Quadrangular-Expand pattern and the foldable pneumatic bellow. Integrated ori
Multiple Sclerosis (MS) lesion segmentation from MR images is important for neuroimaging analysis. MS is diffuse, multifocal, and tend to involve peripheral brain structures such as the white matter, corpus callosum, and brainstem. Recently, U-Net has made great achievements in medical image segment…