Towards Cross-modality Topic Modelling via Deep Topical Correlation Analysis

Abstract

The cross-modality topic detection in social media retains as an open problem mainly due to the difficulty of dealing with modality independence and modality missing. In this paper, we present a novel Deep Topical Correlation Analysis (DTCA) approach, which achieves robust and accurate topic detecting for micro-blogs and handles aforementioned challenges simultaneously. In particular, bidirectional recurrent neural networks and convolutional neural networks are used to learn deep textual and visual features, respectively. Then a Canonical Correlation Analysis based fusion scheme is proposed, which has two innovations to deal with both two problems mentioned above. We further release a large-scale cross-modal twitter dataset for topic detection. Extensive and quantitative evaluations are conducted with comparisons to several state-of-the-arts and alternative approaches on this dataset. Significant performance gains are reported to demonstrate the merits of proposed approach.