MMPlanner: Zero-Shot Multimodal Procedural Planning with Chain-of-Thought Object State Reasoning
Afrina Tabassum, Bin Guo, Xiyao Ma, Hoda Eldardiry, Ismini Lourentzou
Abstract
Multimodal Procedural Planning (MPP) aims to generate step-by-step instructions that combine text and images, with the central challenge of preserving object-state consistency across modalities while producing informative plans. Existing approaches often leverage large language models (LLMs) to refine textual steps; however, visual object-state alignment and systematic evaluation are largely underexplored.We present MMPlanner, a zero-shot MPP framework that introduces Object State Reasoning Chain-of-Thought (OSR-CoT) prompting to explicitly model object-state transitions and generate accurate multimodal plans. To assess plan quality, we design LLM-as-a-judge protocols for planning accuracy and cross-modal alignment, and further propose a visual step-reordering task to measure temporal coherence.Experiments on RecipePlan and WikiPlan show that MMPlanner achieves state-of-the-art performance, improving textual planning by +6.8% , cross-modal alignment by +11.9% , and visual step ordering by +26.7% .
BibTeX
@inproceedings{emnlp2025_mmplannerzerosho,
title = {MMPlanner: Zero-Shot Multimodal Procedural Planning with Chain-of-Thought Object State Reasoning},
author = {Afrina Tabassum and Bin Guo and Xiyao Ma and Hoda Eldardiry and Ismini Lourentzou},
booktitle = {EMNLP 2025},
year = {2025}
}