AnyMAL: An Efficient and Scalable Any-Modality Augmented Language Model
Seungwhan Moon, Andrea Madotto, Zhaojiang Lin, Tushar Nagarajan, Matt Smith, Shashank Jain, Chun-Fu Yeh, Prakash Murugesan
Abstract
We present Any-Modality Augmented Language Model (AnyMAL), a unified model that reasons over diverse input modality signals (i.e. text, image, video, audio, IMU motion sensor), and generates textual responses. AnyMAL inherits the powerful text-based reasoning abilities of the state-of-the-art LLMs including Llama-3 (70B), and converts modality-specific signals to the joint textual space through a pre-trained aligner module.In this paper, we provide details on the optimizations implemented to efficiently scale the training pipeline, and present a comprehensive recipe for model and training configurations. We conduct comprehensive empirical analysis comprising both human and automatic evaluations, and demonstrate state-of-the-art performance on various multimodal tasks compared to industry-leading models – albeit with a relatively small number of trainable parameters.
BibTeX
@inproceedings{moon-etal-2024-anymal,
title = "{A}ny{MAL}: An Efficient and Scalable Any-Modality Augmented Language Model",
author = "Moon, Seungwhan and
Madotto, Andrea and
Lin, Zhaojiang and
Nagarajan, Tushar and
Smith, Matt and
Jain, Shashank and
Yeh, Chun-Fu and
Murugesan, Prakash and
Heidari, Peyman and
Liu, Yue and
Srinet, Kavya and
Damavandi, Babak and
Kumar, Anuj",
editor = "Dernoncourt, Franck and
Preo{\c{t}}iuc-Pietro, Daniel and
Shimorina, Anastasia",
booktitle = "Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track",
month = nov,
year = "2024",
address = "Miami, Florida, US",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2024.emnlp-industry.98/",
doi = "10.18653/v1/2024.emnlp-industry.98",
pages = "1314--1332"
}