INSPECT: A Multimodal Dataset for Pulmonary Embolism Diagnosis and Prognosis
Shih-Cheng Huang, Zepeng Huo, Ethan Steinberg, Chia-Chun Chiang, Matthew P. Lungren, Curtis Langlotz, Serena Yeung, Nigam Shah
Abstract
Synthesizing information from various data sources plays a crucial role in the practice of modern medicine. Current applications of artificial intelligence in medicine often focus on single-modality data due to a lack of publicly available, multimodal medical datasets. To address this limitation, we introduce INSPECT, which contains de-identified longitudinal records from a large cohort of pulmonary embolism (PE) patients, along with ground truth labels for multiple outcomes. INSPECT contains data from 19,402 patients, including CT images, sections of radiology reports, and structured electronic health record (EHR) data (including demographics, diagnoses, procedures, and vitals). Using our provided dataset, we develop and release a benchmark for evaluating several baseline modeling approaches on a variety of important PE related tasks. We evaluate image-only, EHR-only, and fused models. Trained models and the de-identified dataset are made available for non-commercial use under a data use agreement. To the best our knowledge, INSPECT is the largest multimodal dataset for enabling reproducible research on strategies for integrating 3D medical imaging and EHR data.
BibTeX
@inproceedings{
huang2023inspect,
title={{INSPECT}: A Multimodal Dataset for Pulmonary Embolism Diagnosis and Prognosis},
author={Shih-Cheng Huang and Zepeng Huo and Ethan Steinberg and Chia-Chun Chiang and Matthew P. Lungren and Curtis Langlotz and Serena Yeung and Nigam Shah and Jason Alan Fries},
booktitle={Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track},
year={2023},
url={https://openreview.net/forum?id=3sRR2u72oQ}
}