基于空频感知与差分互补的语义驱动实时红外可见光融合

doi:10.3969/j.issn.0255-8297.2026.04.012

应用科学学报 ›› 2026, Vol. 44 ›› Issue (4): 685-700.doi: 10.3969/j.issn.0255-8297.2026.04.012

• 智能视觉感知 • 上一篇

基于空频感知与差分互补的语义驱动实时红外可见光融合

张组军¹, 甘睿², 李威¹, 陈九九¹, 谢宇¹, 熊邦书¹

1. 南昌航空大学信息工程学院, 江西南昌 330063;
2. 江西省交通投资集团有限责任公司, 江西南昌 330013

收稿日期:2026-02-05 发布日期:2026-08-01
通信作者: 熊邦书，教授，研究方向为图像处理，计算机视觉和直升机故障诊断。E-mail:xiongbs@126.com E-mail:xiongbs@126.com
基金资助:
国家自然科学基金(No.62473187)；江西省职业早期青年科技人才项目(20244BCE52091)；江西省教育厅科技项目(GJJ2401015)；江西省图像处理与模式识别重点实验室开放基金(ET202404438)资助

Semantic-Driven Real-Time Infrared and Visible Image Fusion Based on Spatial-Frequency Perception and Differential Complementation

ZHANG Zujun¹, GAN Rui², LI Wei¹, CHEN Jiujiu¹, XIE Yu¹, XIONG Bangshu¹

1. School of Information Engineering, Nanchang Hangkong University, Nanchang 330063, Jiangxi, China;
2. Jiangxi Communications Investment Group Co., Ltd., Nanchang 330013, Jiangxi, China

Received:2026-02-05 Published:2026-08-01

摘要/Abstract

摘要： 针对现有方法全局特征提取受限、跨模态互补特征融合不充分，以及忽视下游视觉任务需求等问题，提出一种基于空频感知与差分互补的语义驱动实时红外可见光融合方法。首先，设计空频双分支感知模块，有效捕捉图像的局部空间细节与全局依赖关系；其次，设计跨模态差分特征互补模块，充分融合不同模态的优势特征；再次，构建语义驱动的联合训练框架，利用语义分割损失引导融合网络保留更多语义信息，从而提高下游高级视觉任务的性能。在RoadScene、MSRS及M3FD公开数据集上的实验结果表明，所提方法与主流方法相比，融合图像的互信息和视觉保真度分别平均提升了3.20%和3.79%；在语义分割任务中，平均交并比提升了2.98%；在目标检测任务中，平均精度均值提升了5.64%。在运行效率方面，处理帧率达到38.99 FPS，满足工程应用的实时性要求。

关键词: 图像融合, 空频双分支感知, 跨模态差分特征互补, 语义驱动

Abstract: To address the limitations of existing methods, such as limited global feature extraction, insufficient fusion of cross-modal complementary features, and the neglect of downstream visual task requirements, a semantic-driven real-time infrared and visible image fusion method based on spatial-frequency perception and differential complementation is proposed. First, a spatial-frequency dual-branch perception module was designed to effectively capture the local spatial details and global dependencies of the image. Second, a cross-modality differential feature complementation module was designed to fully integrate the advantageous features of different modalities. Furthermore, a semantic-driven joint training framework was constructed, using semantic segmentation loss to guide the fusion network to retain more semantic information, thereby improving the performance of downstream advanced visual tasks. Experimental results on the public datasets RoadScene, MSRS, and M3FD show that, compared with mainstream methods, the mutual information and visual fidelity of the fused images are improved by an average of 3.20% and 3.79%, respectively, by the proposed method; in the semantic segmentation task, the mean intersection over union is improved by 2.98%; in the object detection task, the mean average precision is improved by 5.64%. In terms of running efficiency, the processing frame rate reaches 38.99 FPS, meeting the real-time requirements of engineering applications.

Key words: image fusion, spatial-frequency dual-branch perception, cross-modality differential feature complementation, semantic drive

中图分类号:

TP391

张组军, 甘睿, 李威, 陈九九, 谢宇, 熊邦书. 基于空频感知与差分互补的语义驱动实时红外可见光融合[J]. 应用科学学报, 2026, 44(4): 685-700.

ZHANG Zujun, GAN Rui, LI Wei, CHEN Jiujiu, XIE Yu, XIONG Bangshu. Semantic-Driven Real-Time Infrared and Visible Image Fusion Based on Spatial-Frequency Perception and Differential Complementation[J]. Journal of Applied Sciences, 2026, 44(4): 685-700.

参考文献

[1] 金安安, 李祥, 张丽, 等. 基于NSCT与压缩感知的红外影像融合[J]. 应用科学学报, 2022, 40(1): 80-92. Jin A A, Li X, Zhang L, et al. Infrared image fusion based on NSCT and compressed sensing [J]. Journal of Applied Sciences, 2022, 40(1): 80-92. (in Chinese)
[2] Li Y, Li D, Fang S, et al. Multimodal image fusion network with prior-guided dynamic degradation removal for extreme environment perception [J]. Scientific Reports, 2025, 15(1): 40530- 40551.
[3] Li S, Yang B, Hu J. Performance comparison of different multi-resolution transforms for image fusion [J]. Information Fusion, 2011, 12(2): 74-84.
[4] Li H, Wu X J. Multi-focus image fusion using dictionary learning and low-rank representation [C] //International Conference on Image and Graphics, 2017: 675-686.
[5] Li S, Yin H, Fang L. Group-sparse representation with dictionary learning for medical image denoising and fusion [J]. IEEE Transactions on Biomedical Engineering, 2012, 59(12): 3450-3459.
[6] Ma J, Tang L, Xu M, et al. STDFusionNet: an infrared and visible image fusion network based on salient target detection [J]. IEEE Transactions on Instrumentation and Measurement, 2021, 70: 1-13.
[7] Xu J, Zhang J. Infrared and visible image fusion based on Haar wavelet downsampling and multi-scale feature aggregation [J]. Signal, Image and Video Processing, 2025, 19(14): 1200- 1208.
[8] Duan X, Liu Y, Yang Z, et al. CCFusion: a novel channel-convolutional neural network for infrared and visible image fusion [J]. IEEE Access, 2026, 14: 35396-35409.
[9] Tang W, He F, Liu Y. ITFuse: an interactive transformer for infrared and visible image fusion [J]. Pattern Recognition, 2024, 156: 110822-110834.
[10] 华怡坦, 黄影平, 过文昊. 基于CNN和Transformer点云图像融合的道路检测[J]. 应用科学学报, 2024, 42(4): 695-708. Hua Y T, Huang Y P, Guo W H. Fusion of point-cloud and image for road segmentation using CNN and Transformer [J]. Journal of Applied Sciences, 2024, 42(4): 695-708. (in Chinese)
[11] Hou Q, Lu C Z, Cheng M M, et al. Conv2Former: a simple transformer-style convnet for visual recognition [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024, 46(12): 8274-8283.
[12] Li H, Yang Z, Zhang Y, et al. MulFS-CAP: multimodal fusion-supervised cross-modality alignment perception for unregistered infrared-visible image fusion [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025, 47(5): 3673-3690.
[13] Yang Z, Zhang Y, Li H, et al. Instruction-driven fusion of Infrared-visible images: tailoring for diverse downstream tasks [J]. Information Fusion, 2025, 121: 103148-103162.
[14] Tang L, Yuan J, Ma J. Image fusion in the loop of high-level vision tasks: a semantic-aware real-time infrared and visible image fusion network [J]. Information Fusion, 2022, 82: 28-42.
[15] Bao W, Feng Z, Du Y, et al. A dual-modal semantic guidance and differential feature complementation fusion method for infrared and visible image [J]. Signal, Image and Video Processing, 2025, 19(2): 150-164.
[16] Yu C, Wang J, Peng C, et al. BiSeNet: bilateral segmentation network for real-time semantic segmentation [C] //European Conference on Computer Vision, 2018: 325-341.
[17] Xu H, Ma J, Jiang J, et al. U2Fusion: a unified unsupervised image fusion network [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, 44(1): 502-518.
[18] Tang L, Yuan J, Zhang H, et al. PIAFusion: a progressive infrared and visible image fusion network based on illumination aware [J]. Information Fusion, 2022, 83: 79-92.
[19] Liu J, Fan X, Huang Z, et al. Target-aware dual adversarial learning and a multi-scenario multimodality benchmark to fuse infrared and visible for object detection [C] //IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022: 5802-5811.
[20] Zhang W, Sun H, Zhou B. TBRAFusion: infrared and visible image fusion based on two-branch residual attention transformer [J]. Electronic Research Archive, 2025, 33(1): 158-180.
[21] Zhao Z, Bai H, Zhang J, et al. CDDFuse: correlation-driven dual-branch feature decomposition for multi-modality image fusion [C] //IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023: 5906-5916.
[22] Li H, Wu X J. CrossFuse: a novel cross attention mechanism based infrared and visible image fusion approach [J]. Information Fusion, 2024, 103: 102147-102158.
[23] Xie X, Cui Y, Tan T, et al. FusionMamba: dynamic feature enhancement for multimodal image fusion with mamba [J]. Visual Intelligence, 2024, 2(1): 37-54.
[24] Tang L, Yan Q, Xiang X, et al. C2RF: bridging multi-modal image registration and fusion via commonality mining and contrastive learning [J]. International Journal of Computer Vision, 2025: 1-19.

基于空频感知与差分互补的语义驱动实时红外可见光融合

Semantic-Driven Real-Time Infrared and Visible Image Fusion Based on Spatial-Frequency Perception and Differential Complementation

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 5

编辑推荐

Metrics

本文评价

[1]	金安安, 李祥, 张丽, 熊卿智. 基于NSCT与压缩感知的红外影像融合[J]. 应用科学学报, 2022, 40(1): 80-92.
[2]	楼建强, 戴文战, 李俊峰. 基于非下采样Contourlet变换的自适应医学图像融合算法[J]. 应用科学学报, 2017, 35(6): 763-774.
[3]	刘洋, 徐洪平, 易航, 施清平, 夏伟强, 康健. 基于有约束CNMF的遥感高光谱多光谱图像融合[J]. 应用科学学报, 2016, 34(6): 651-660.
[4]	唐振华, 梁聪, 区骋, 黄旭方, 覃团发. 基于DCT域纹理特征的多聚焦图像融合[J]. 应用科学学报, 2015, 33(6): 628-636.
[5]	潘瑜，王静，孙权森，夏德深. 结合图像质量评价参数的多尺度分解融合策略[J]. 应用科学学报, 2011, 29(2): 159-168.