Multimodal Human Activity Recognition with a Large Language Model for Enhanced Human-Robot Interaction

Abstract

This paper presents a novel framework for Human Activity Recognition (HAR) by unifying all sensor streams, visual, audio, and inertial, into a single textual domain, enabling the direct application of GPT-3 for multimodal data classification. Unlike traditional pipelines that use dedicated encoders for each modality, we show that converting sensor outputs into text tokens offers both simplicity and a powerful proof-of-concept for large language models (LLMs). To further boost performance, we introduce a composite loss function combining cross-entropy, Kullback-Leibler divergence, total variation, and multimodal consistency terms, ensuring both temporal smoothness and cross-modal alignment. We conduct extensive experiments on the CMU-MMAC dataset, achieving up to 98% accuracy and significantly outperforming baseline methods. We also demonstrate robustness under missing sensor streams via partial tokenization, maintaining strong performance despite sensor failures. These results highlight the potential of LLM-driven HAR for enhanced human-robot interaction in real-world scenarios, and pave the way for broader multimodal applications of next-generation language models.