Language-Driven Multi-Task Manipulation With Action-Mask-Enhanced Multimodal Learning

Abstract

For long-horizon multi-task robotic manipulation, hierarchical approaches provide an effective way to combine high-level language-based task planning with low-level vision-language based sub-task execution. Then, we propose a framework that integrates a two-stage task planner with a multimodal low-level action planner incorporating an explicit action-mask policy. At high level, a Vision-Language Model (VLM) first perceives object and scene information from observations, and a Large Language Model (LLM) then reasons over this together with a task library and human instruction to generate a textual task plan. This two-stage design mitigates modality bias between perception and planning. At low level, an asymmetric multimodal encoder, SigLIP2 with Weight-Decomposed Low-Rank Adaptation (DoRA) for text and multi-view ResNets for vision, feeds into an Action Chunking with Transformer (ACT)-based policy enhanced by Temperature-Scaled Spatial Attention and Bidirectional Cross-Attention for language-vision fusion. We further introduce an explicit action-mask policy that jointly predicts actions and their validity, enabling real-time sub-task termination detection and robust switching across multiple sub-tasks without additional inference overhead. Experiments on weighing and multi-object manipulation tasks demonstrate performance, with ablation studies validating the contribution of each component. Generalization experiments under out-of-distribution conditions highlight both the strengths and limitations. Deployment on a distinct dual-arm robotic platform in a new scenario validates transferability.