Pengxiang Cheng — accepted AI-conference papers

2025

Improving Instruct Models for Free: A Study on Partial Adaptation

Ozan Irsoy, Pengxiang Cheng, Jennifer L Chen, Daniel Preotiuc-Pietro, Shiyue Zhang, Duccio Pappadopulo

EMNLP 2025

Instruct models, obtained from various instruction tuning or post-training steps, are commonly deemed superior and more usable than their base counterpart. While the model gains instruction following ability, instruction tun- ing may lead to forgetting the knowledge from pre-training or it may encou

Cited by 0 Source ↗PDF ↗code ↗Scholar ↗

2023

Dataless Knowledge Fusion by Merging Weights of Language Models

Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietro, Pengxiang Cheng

ICLR 2023poster

Fine-tuning pre-trained language models has become the prevalent paradigm for building downstream NLP models. Oftentimes fine-tuned models are readily available but their training data is not, due to data privacy or intellectual property concerns. This creates a barrier to fusing knowledge across in…

Cited by 201 Source ↗PDF ↗code ↗Scholar ↗

2023

Overcoming Catastrophic Forgetting in Massively Multilingual Continual Learning

Genta Winata, Lingjue Xie, Karthik Radhakrishnan, Shijie Wu, Xisen Jin, Pengxiang Cheng +2

ACL 2023findings

Real-life multilingual systems should be able to efficiently incorporate new languages as data distributions fed to the system evolve and shift over time. To do this, systems need to handle the issue of catastrophic forgetting, where the model performance drops for languages or tasks seen further in…

Cited by 22 Source ↗PDF ↗Scholar ↗