AIConfPaper
← Search

Rabeeh Karimi Mahabadi

6 accepted papers

2026

Nemotron-CC-Math: A 133 Billion-Token-Scale High Quality Math Pretraining Dataset

Rabeeh Karimi mahabadi, Sanjeev Satheesh, Shrimai Prabhumoye, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro
ICLR 2026poster

Pretraining large language models (LLMs) on high-quality, structured data such as mathematics and code substantially enhances reasoning capabilities. However, existing math-focused datasets built from Common Crawl suffer from degraded quality due to brittle extraction heuristics, lossy HTML-to-text…

Cited by 0Source ↗Scholar ↗
2022

Prompt-free and Efficient Few-shot Learning with Language Models

Rabeeh Karimi Mahabadi, Luke Zettlemoyer, James Henderson, Lambert Mathias, Marzieh Saeidi, Veselin Stoyanov +1
ACL 2022long

Current methods for few-shot fine-tuning of pretrained masked language models (PLMs) require carefully engineered prompts and verbalizers for each new task to convert examples into a cloze-format that the PLM can score. In this work, we propose Perfect, a simple and efficient method for few-shot fin…

Cited by 78Source ↗PDF ↗code ↗Scholar ↗
2021

Compacter: Efficient Low-Rank Hypercomplex Adapter Layers

Rabeeh Karimi mahabadi, James Henderson, Sebastian Ruder
NeurIPS 2021poster

Adapting large-scale pretrained language models to downstream tasks via fine-tuning is the standard method for achieving state-of-the-art performance on NLP benchmarks. However, fine-tuning all weights of models with millions or billions of parameters is sample-inefficient, unstable in low-resource…

Cited by 500Source ↗PDF ↗code ↗Scholar ↗
2021

Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks

Rabeeh Karimi Mahabadi, Sebastian Ruder, Mostafa Dehghani, James Henderson
ACL 2021long

State-of-the-art parameter-efficient fine-tuning methods rely on introducing adapter modules between the layers of a pretrained language model. However, such modules are trained separately for each task and thus do not enable sharing information across tasks. In this paper, we show that we can learn…

Cited by 342Source ↗PDF ↗code ↗Scholar ↗
2021

Variational Information Bottleneck for Effective Low-Resource Fine-Tuning

Rabeeh Karimi mahabadi, Yonatan Belinkov, James Henderson
ICLR 2021poster

While large-scale pretrained language models have obtained impressive results when fine-tuned on a wide variety of tasks, they still often suffer from overfitting in low-resource scenarios. Since such models are general-purpose feature extractors, many of these features are inevitably irrelevant for…

Cited by 66Source ↗PDF ↗code ↗Scholar ↗