Large Language Model-Based Robot Task Planning from Voice Command Transcriptions

Abstract

One of the primary challenges in building a General Purpose Service Robot (GPSR), i.e. a robot capable of executing generic human commands, lies in acting upon natural language instructions. These instructions often contain speech recognition errors and incomplete information, complicating the extraction of clear goals and the formulation of an efficient and effective action plan. This work presents a pipeline that leverages a Large Language Model to directly translate instruction transcripts into coherent action plans. The pipeline also integrates environmental context into the model’s input, allowing for the generation of more efficient and context-aware plans. The system’s performance was evaluated using a simulator based on generalized stochastic Petri Nets, achieving a success rate of around 55% on the ALFRED dataset, even in unseen environments. The entire pipeline was also successfully deployed at RoboCup 2024 in Eindhoven, where it secured second place in the GPSR task. The code, dataset, and models are available at https://github.com/socrob/llm_gpsr.