Prompt-To-State Stable Vision-Language MPC for Approximated Neural Network Dynamics a Case Study on Soft Robot Control
Nicotra Emanuele, James J. Davies, Kefan Zhu, Sharma Bibhu, Adrienne Ji, Phuoc Thien Phan, Hung Manh La, Nigel Hamilton Lovell
Abstract
The integration of large-scale foundation models in control loops has shown strong potential for executing complex tasks directly from natural language inputs. However, achieving stability and real-time performance remains a sig- nifi cant challenge, particularly for systems with hard-to-model dynamics. In this paper, we introduce Prompt-to-State Stability (PSS) and propose the Prompt-to-State Stable Vision-Language Model Predictive Control (PSS-VLMPC) framework, which couples a vision-language model (VLM) with a robust model predictive control (MPC) scheme. The VLM interprets user commands and visual feedback, converting them into control- relevant parameters for the MPC. System dynamics are fully learned by a neural network and then approximated to enable real-time MPC performance. Building on prediction error bounds, we provide rigorous closed-loop stability guarantee and validate the effectiveness of PSS-VLMPC through both simulations and real-world experiments on a soft continuum robot, demonstrating its ability to robustly execute tasks from natural language instructions.