Memory-QA: Answering Recall Questions Based on Multimodal Memories
Hongda Jiang, Xinyuan Zhang, Siddhant Garg, Rishab Arora, Shiun-Zu Kuo, Jiayang Xu, Aaron Colak, Xin Luna Dong
Abstract
We introduce Memory-QA, a novel real-world task that involves answering recall questions about visual content from previously stored multimodal memories. This task poses unique challenges, including the creation of task-oriented memories, the effective utilization of temporal and location information within memories, and the ability to draw upon multiple memories to answer a recall question. To address these challenges, we propose a comprehensive pipeline, Pensieve, integrating memory-specific augmentation, time- and location-aware multi-signal retrieval, and multi-memory QA fine-tuning. We created a multimodal benchmark to illustrate various real challenges in this task, and show the superior performance of Pensieve over state-of-the-art solutions (up to +14% on QA accuracy).
BibTeX
@inproceedings{emnlp2025_memoryqaanswerin,
title = {Memory-QA: Answering Recall Questions Based on Multimodal Memories},
author = {Hongda Jiang and Xinyuan Zhang and Siddhant Garg and Rishab Arora and Shiun-Zu Kuo and Jiayang Xu and Aaron Colak and Xin Luna Dong},
booktitle = {EMNLP 2025},
year = {2025}
}