INDUS: Effective and Efficient Language Models for Scientific Applications
Bishwaranjan Bhattacharjee, Aashka Trivedi, Masayasu Muraoka, Muthukumaran Ramasubramanian, Takuma Udagawa, Iksha Gurung, Nishan Pantha, Rong Zhang
Abstract
Large language models (LLMs) trained on general domain corpora showed remarkable results on natural language processing (NLP) tasks. However, previous research demonstrated LLMs trained using domain-focused corpora perform better on specialized tasks. Inspired by this insight, we developed INDUS, a comprehensive suite of LLMs tailored for the closely-related domains of Earth science, biology, physics, heliophysics, planetary sciences and astrophysics, and trained using curated scientific corpora drawn from diverse data sources. The suite of models include: (1) an encoder model trained using domain-specific vocabulary and corpora to address NLP tasks, (2) a contrastive-learning based text embedding model trained using a diverse set of datasets to address information retrieval tasks and (3) smaller versions of these models created using knowledge distillation for applications which have latency or resource constraints. We also created three new scientific benchmark datasets, Climate-Change NER (entity-recognition), NASA-QA (extractive QA) and NASA-IR (IR) to accelerate research in these multi-disciplinary fields. We show that our models outperform both general-purpose (RoBERTa) and domain- specific (SciBERT) encoders on these new tasks as well as existing tasks in the domains of interest. Furthermore, we demonstrate the use of these models in two industrial settings- as a retrieval model for large-scale vector search applications and in automatic content tagging systems.
BibTeX
@inproceedings{bhattacharjee-etal-2024-indus,
title = "{INDUS}: Effective and Efficient Language Models for Scientific Applications",
author = "Bhattacharjee, Bishwaranjan and
Trivedi, Aashka and
Muraoka, Masayasu and
Ramasubramanian, Muthukumaran and
Udagawa, Takuma and
Gurung, Iksha and
Pantha, Nishan and
Zhang, Rong and
Dandala, Bharath and
Ramachandran, Rahul and
Maskey, Manil and
Bugbee, Kaylin and
Little, Michael M. and
Fancher, Elizabeth and
Gerasimov, Irina and
Mehrabian, Armin and
Sanders, Lauren and
Costes, Sylvain V. and
Blanco-Cuaresma, Sergi and
Lockhart, Kelly and
Allen, Thomas and
Grezes, Felix and
Ansdell, Megan and
Accomazzi, Alberto and
El-Kurdi, Yousef and
Wertheimer, Davis and
Pfitzmann, Birgit and
Berrospi Ramis, Cesar and
Dolfi, Michele and
De Lima, Rafael Teixeira and
Vagenas, Panagiotis and
Mukkavilli, S. Karthik and
Staar, Peter W. J. and
Vahidinia, Sanaz and
McGranaghan, Ryan and
Lee, Tsengdar J.",
editor = "Dernoncourt, Franck and
Preo{\c{t}}iuc-Pietro, Daniel and
Shimorina, Anastasia",
booktitle = "Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track",
month = nov,
year = "2024",
address = "Miami, Florida, US",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2024.emnlp-industry.9/",
doi = "10.18653/v1/2024.emnlp-industry.9",
pages = "98--112"
}