← Search

Shan Jiang

11 accepted papers

2025

Hidden in Plain Sight: Reasoning in Underspecified and Misspecified Scenarios for Multimodal LLMs

EMNLP 2025

Multimodal large language models (MLLMs) are increasingly deployed in open-ended, real-world environments where inputs are messy, underspecified, and not always trustworthy. Unlike curated benchmarks, these settings frequently involve instructions that reference missing objects or contradictory fact

Cited by 0SourcePDFScholar
2025

Multimodal Inconsistency Reasoning (MMIR): A New Benchmark for Multimodal Reasoning Models

ACL 2025finding

Existing Multimodal Large Language Models (MLLMs) are predominantly trained and tested on consistent visual-textual inputs, leaving open the question of whether they can handle inconsistencies in real-world, layout-rich content. To bridge this gap, we propose the Multimodal Inconsistency Reasoning (…

2024

Muffin or Chihuahua? Challenging Multimodal Large Language Models with Multipanel VQA

ACL 2024long

Multipanel images, commonly seen as web screenshots, posters, etc., pervade our daily lives. These images, characterized by their composition of multiple subfigures in distinct layouts, effectively convey information to people. Toward building advanced multimodal AI applications, such as agents that…

Cited by 20SourcePDFScholar
2024

PRoDeliberation: Parallel Robust Deliberation for End-to-End Spoken Language Understanding

EMNLP 2024finding

Spoken Language Understanding (SLU) is a critical component of voice assistants; it consists of converting speech to semantic parses for task execution. Previous works have explored end-to-end models to improve the quality and robustness of SLU models with Deliberation, however these models have rem…

Cited by 0SourcePDFScholar
2024

Read Anywhere Pointed: Layout-aware GUI Screen Reading with Tree-of-Lens Grounding

EMNLP 2024main

Graphical User Interfaces (GUIs) are central to our interaction with digital devices and growing efforts have been made to build models for various GUI understanding tasks. However, these efforts largely overlook an important GUI-referring task: screen reading based on user-indicated points, which w…

2023

ICASSP 2023 Spoken Language Understanding Grand Challenge

ICASSP 2023accepted

Spoken language understanding (SLU) is a important field between the Speech and NLP community focused on converting a users’ speech utterance into an executable semantic parse. In order to facilitate open research in this space, we introduce the 1st Spoken Language Understanding challenge hosted at…

Cited by 0SourceScholar
2023

Is Weakly-Supervised Action Segmentation Ready for Human-Robot Interaction? No, Let's Improve It with Action-Union Learning

IROS 2023poster

Action segmentation plays an important role in enabling robots to automatically understand human activities. To train the action recognition model, while obtaining action labels for all frames is costly, annotating timestamp labels for weak supervision is cost-effective. However, existing methods ma…

Cited by 3SourceScholar