RDMM: Enhancing Household Robotics with On-Device Contextual Memory and Decision Making

Abstract

Large language models (LLMs) represent a significant advancement in integrating physical robots with AI-driven systems. In this research, we present a framework that leverages Robotics Decision-Making Models (RDMM) for decision-making in domain-specific contexts, enhancing robotic autonomy. This framework incorporates agent-specific knowledge representation, allowing robots to recall and utilize their capabilities and past experiences for improved decision-making. Unlike other approaches, our method prioritizes real-time, on-device solutions, successfully operating on hardware with as little as 8GB of memory. The framework integrates visual perception models, providing robots with a better understanding of their environment. Additionally, real-time speech recognition capabilities are included, improving the human-robot interaction experience. Experimental results show that the RDMM framework achieves planning accuracy of 93%. Furthermore, we introduce a novel dataset consisting of 27k planning instances and 1.3k annotated text-image samples, specifically curated from real-world robotic tasks in competition scenarios. The framework, benchmarks, datasets, and models developed in this work are publicly available on our project website at https://github.com/shadynasrat/RDMM.