Hyperbolic Transformers with LLMs for Multimodal Human Activity Recognition

Abstract

Human Activity Recognition (HAR) plays a crucial role in applications such as healthcare, smart environments, and human-robot interaction. This study proposes a novel Hyperbolic optimization strategy to improve model generalization by leveraging the geometric structure of the parameter space. To evaluate its effectiveness while also benefiting from the capabilities of modern sequence models in capturing long-range dependencies and multimodal interactions, the loss function is integrated into Transformer and GPT-2 models, fine-tuned on both unimodal (UCI-HAR, Opportunity) and multimodal (UTD-MHAD, NTU RGB+D) datasets. Unlike prior work that typically leverages only one to three modalities, this study utilizes all available modalities—RGB, depth, skeleton, and inertial—for multimodal evaluation. The Transformer achieves 98.26% and 93.40% accuracy on UCI-HAR and Opportunity, respectively, and 99.08% and 89.93% on UTD-MHAD and NTU RGB+D. GPT-2 also performs competitively, achieving 86.33% and 83.57% on the unimodal datasets, and 83.23% and 86.51% on the multimodal ones. These results highlight the potential of hyperbolic optimization for HAR across diverse sensor modalities and architectures.