为进一步提升谣言检测的有效性和稳定性，提出了一种融合图文特征自适应抗噪与语义修补的多模态谣言检测方法。首先，设计预训练编码、动态池化以及多头增强的三级处理流程，将谣言原文与评论文本编码成语义向量。然后，构建视觉特征提取和光学字符特征提取两个并行模块，将谣言图像及图像中的显式文本编码成互补增强的图像向量。最后，通过自适应门控和动态交叉注意力机制过滤噪声并增强语义，实现图文信息的局部对齐与全局整合。实验结果表明，相较于对比算法，所提方法能够有效捕捉图文信息的深层关联，提升了谣言检测结果的可信性和实用性。
To further improve the effectiveness and stability of rumor detection, a multimodal rumor detection method was proposed that integrated image and text features with adaptive noise resistance and semantic restoration. First, a three-stage processing pipeline consisting of pretrained encoding, dynamic pooling, and multi-head enhancement was designed to encode rumor-related texts and comments into semantic vectors. Then,two parallel modules were constructed: one for visual feature extraction and the other for optical character feature extraction. These modules encoded rumor images and explicit text within them into complementary enhanced image vectors. Finally, adaptive gating and dynamic cross-attention mechanisms were used to filter noise and enhance semantics,achieving local alignment and global integration of image and text information. Experimental results show that, compared with baseline algorithms, the proposed method can effectively capture deep correlations between image and text information and improve credibility and practicality of rumor detection results.
[1] Chen Y X, Li D S, Zhang P, et al. Cross-modal ambiguity learning for multimodal fake news detection [C]//ACM Web Conference, 2022: 2897-2905.
[2] Jing J, Wu H C, Sun J, et al. Multimodal fake news detection via progressive fusion networks [J]. Information Processing and Management, 2023, 60(1): 103120-103133.
[3] Singhal S, Pandey T, Mrig S, et al. Leveraging intra and inter modality relationship for multimodal fake news detection [C]//Companion Web Conference, 2022: 726-734.
[4] 蒋超, 朱学芳. 基于模态融合增强的谣言检测研究[J]. 数据分析与知识发现, 2025, 9(7): 26-37. Jiang C, Zhu X F. Research on rumour detection based on modal fusion enhancement [J]. Data Analysis and Knowledge Discovery, 2025, 9(7): 26-37. (in Chinese)
[5] Wang L Z, Zhang C, Xu H B, et al. Cross-modal contrastive learning for multimodal fake news detection [C]//31st International Conference on Multimedia, 2023: 5696-5704.
[6] Meel P, Vishwakarma D K. Han, image captioning, and forensics ensemble multimodal fake news detection [J]. Information Sciences, 2021, 567: 23-41.
[7] Song C G, Ning N W, Zhang Y L, et al. A multimodal fake news detection model based on crossmodal attention residual and multichannel convolutional neural networks [J]. Information Processing and Management, 2021, 58(1): 102437-102445.
[8] Kumari R, Ekbal A. AMFB: attention based multimodal factorized bilinear pooling for multimodal fake news detection [J]. Expert Systems with Applications, 2021, 184(1): 115412-115424.
[9] Segura-Bedmar I, Alonso-Bartolome S. Multimodal fake news detection [J]. Information, 2022, 13(6): 284-300.
[10] Hua J H, Cui X D, Li X H, et al. Multimodal fake news detection through data augmentationbased contrastive learning [J]. Applied Soft Computing, 2023, 136: 110125-110134.
[11] Giachanou A, Zhang G B, Rosso P. Multimodal fake news detection with textual, visual and semantic information [C]//Text, Speech, and Dialogue, 2020: 30-38.
[12] Liu F, Xu G H, Wu Q, et al. Cascade reasoning network for text-based visual question answering [C]//28th International Conference on Multimedia, 2020: 4060-4069.
[13] Han W, Huang H T, Han T. Finding the evidence: localization-aware answer prediction for text visual question answering [C]//28th International Conference on Computational Linguistics, 2020: 3118-3131.
[14] Wang C Y, Zhang M, Shi F, et al. A hybrid multimodal data fusion-based method for identifying gambling websites [J]. Electronics, 2022, 11(16): 1-18.
[15] Lin Z H, Xie J Y, Li Q. Multi-modal news event detection with external knowledge [J]. Information Processing and Management, 2024, 61(3): 103697-103714.
[16] 刘先博, 向澳, 杜彦辉. 融合图文多粒度情感特征的多模态谣言检测方法[J]. 计算机科学与探索, 2025, 19(4): 1021-1035. Liu X B, Xiang A, Du Y H. Multimodal rumor detection method based on multi-granularity emotional features of image-text [J]. Journal of Frontiers of Computer Science and Technology, 2025, 19(4): 1021-1035. (in Chinese)
[17] Zhang L W B, Wilson R, Sumner M, et al. Advanced multimodal fusion method for very short-term solar irradiance forecasting using sky images and meteorological data: a gate and transformer mechanism approach [J]. Renewable Energy, 2023, 216: 118952-118959.
[18] 黄学坚, 马廷淮, 荣欢, 等. 融合外部知识与证据的场景图注意力网络多模态谣言检测[J]. 计算机学报, 2025, 48(9): 2159-2180. Huang X J, Ma T H, Rong H, et al. Multi-modal rumor detection with scene graph attention networks integrating external knowledge and evidence [J]. Chinese Journal of Computers, 2025, 48(9): 2159-2180. (in Chinese)
[19] Kumar A, Vepa J. Gated mechanism for attention based multimodal sentiment analysis [C]//45th IEEE International Conference on Acoustics, Speech and Signal Processing, 2020: 4477-4481.
[20] Jin Z W, Cao J, Guo H, et al. Multimodal fusion with recurrent neural networks for rumor detection on microblogs [C]//25th ACM International Conference on Multimedia, 2017: 795-816.
[21] Wu Y, Zhan P W, Zhang Y J, et al. Multimodal fusion with co-attention networks for fake news detection [C]//Findings of the Association for Computational Linguistics, 2021: 2560-2569.
[22] Peng L W, Jian S L, Li D S, et al. MRML: multimodal rumor detection by deep metric learning [C]//48th IEEE International Conference on Acoustics, Speech and Signal Processing, 2023: 10096188-10096193.
[23] Xu F, Zeng L, Huang Q, et al. Hierarchical graph attention networks for multi-modal rumor detection on social media [J]. Neurocomputing, 2024, 569: 127112-127123.
