针对低照度环境下获取的图像与颜色失真问题，该文结合了通道注意力和空间注意力的机制，提出了一种基于颜色还原Transformer网络（color restoraration Transformer networks,CRTNet）的低照度图像增强方法。CRTNet由颜色注意力模块（color attention module,CAM）、颜色映射模块（color map module,CMM）和顺序增强结构组成。首先，CAM分为颜色通道注意力模块和颜色空间注意力模块两部分，利用Transformer的全局信息捕捉能力，颜色通道注意力模块关注有意义的颜色通道并赋予更高权重，生成通道注意力向量，颜色空间注意力模块使用三层卷积结构，关注高维空间中的空间细节信息，生成空间注意力权重图；其次，CMM通过线性拟合过程提取图像高维特征，对64D空间中的特征进行通道和空间两个维度的缩放和平移获得图像全局信息和细节信息，并与原始图像特征相结合，补充原始图像特征中颜色、亮度、对比度和细节等信息，实现颜色增强；最后，采用顺序增强结构，将CMM的输出作为输入重复进行3次CAM和CMM操作，以拟合更高阶的函数映射，实现低照度图像的有效增强。对公共数据集的实验和用户研究表明，所提方法在定量测量、细节与颜色复原方面取得了最好的结果。
A low light image enhancement method based on color restoration transformer networks (CRTNet) is proposed to address the issue of image and color distortion in low light environments. This method combines channel attention and spatial attention mechanisms. CRTNet consists of a color attention module (CAM), a color map module (CMM), and a sequential enhancement structure. Firstly, CAM is divided into two parts: color channel attention module and color space attention module. Utilizing the global information capture capability of Transformer, the color channel attention module emphasizes meaningful color channels by assigning higher weights to generate channel attention vectors. The color space attention module uses a three-layer convolution structure, focuses on spatial details in high-dimensional space and generates spatial attention weight map. Secondly, CMM extracts high-dimensional image features through a linear fitting process, scaling and shifting these features in the 64D space across both channel and spatial dimensions to obtain global and detail image information. By combining with the original image features, it supplements the color, brightness, contrast, and detail information in the original image features to achieve color enhancement. Finally, a sequential enhancement structure is adopted to repeat CAM and CMM operations three times with the output of CMM serving as input, in order to fit higher-order function mappings and effectively enhance low light images. Experiments results and user studies on public datasets demonstrate that the proposed method outperforms existing approaches in quantitative measurement, detail and color restoration.
[1] Cheng H D, Shi X J. A simple and effective histogram equalization approach to image enhancement [J]. Digital Signal Processing, 2004, 14(2): 158-170.
[2] Jobson D J, Rahman Z, Woodell G A. A multiscale retinex for bridging the gap between color images and the human observation of scenes [J]. IEEE Transactions on Image Processing, 1997, 6(7): 965-976.
[3] Guo X J, Li Y, Ling H B. LIME: low-light image enhancement via illumination map estimation [J]. IEEE Transactions on Image Processing, 2017, 26(2): 982-993.
[4] Zhao Z J, Xiong B S, Wang L, et al. RetinexDIP: a unified deep framework for low-light image enhancement [J]. IEEE Transactions on Circuits and Systems for Video Technology, 2022, 32(3): 1076-1088.
[5] Liu R S, Ma L, Zhang J A, et al. Retinex-inspired unrolling with cooperative prior architecture search for low-light image enhancement [C]//IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021: 10556-10565.
[6] Wu W H, Weng J, Zhang P P, et al. URetinex-net: retinex-based deep unfolding network for low-light image enhancement [C]//IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022: 5891-5900.
[7] Feng W, Wu G M, Zhou S Q, et al. Low-light image enhancement based on Retinex-Net with color restoration [J]. Applied Optics, 2023, 62(25): 6577-6584.
[8] Dong X, Pang Y A, Wen J G. Fast efficient algorithm for enhancement of low lighting video [DB/OL]. 2010[2023-11-15]. https://dl.acm.org/doi/10.1145/1836845.1836920.
[9] Park D, Kim M, Ku B, et al. Image enhancement for extremely low light conditions [C]//11th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), 2014: 307-312.
[10] Yang Y, Wang C Y, Liu R S, et al. Self-augmented unpaired image dehazing via density and depth decomposition [C]//IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022: 2027-2036.
[11] Jiang Y F, Gong X Y, Liu D, et al. EnlightenGAN: deep light enhancement without paired supervision [J]. IEEE Transactions on Image Processing, 2021, 30: 2340-2349.
[12] Zhang Y, Di X G, Zhang B, et al. Self-supervised low light image enhancement and denoising [DB/OL]. 2021[2023-11-15]. http://arxiv.org/abs/2103.00832.
[13] Wang Y F, Wan R J, Yang W H, et al. Low-light image enhancement with normalizing flow [J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2022, 36(3): 2604-2612.
[14] Liang D, Li L, Wei M Q, et al. Semantically contrastive learning for low-light image enhancement [J]. AAAI Conference on Artificial Intelligence, 2022, 36(2): 1555-1563.
[15] Bychkovsky V, Paris S, Chan E, et al. Learning photographic global tonal adjustment with a database of input/output image pairs [C]//IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2011: 97-104.
[16] Zeng H, Cai J R, Li L D, et al. Learning image-adaptive 3D lookup tables for high performance photo enhancement in real-time [J].IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44(4): 2058-2073.
[17] He J W, Liu Y H, Qiao Y, et al. Conditional sequential modulation for efficient global image retouching [C]//European Conference on Computer Vision, 2020: 679-695.
[18] Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need [C]// 31st International Conference on Neural Information Processing Systems. Red Hook, 2017: 6000-6010.
[19] Dosovitskiy A, Beyer L, Kolesnikov A, et al. An image is worth 16x16 words: transformers for image recognition at scale [DB/OL]. 2020[2023-11-15]. http://arxiv.org/abs/2010.11929.
[20] Cui Z T, Li K C, Gu L, et al. You only need 90K parameters to adapt light: a light weight transformer for image enhancement and exposure correction [DB/OL]. 2022[2023-11- 15]. http://arxiv.org/abs/2205.14871.
[21] Cai Y H, Bian H, Lin J, et al. Retinexformer: one-stage retinex-based transformer for low-light image enhancement [DB/OL]. 2023[2023-11-15]. http://arxiv.org/abs/2303.06705.
[22] Guo C L, Li C Y, Guo J C, et al. Zero-reference deep curve estimation for low-light image enhancement [C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020: 1777-1786.
[23] Ma L, Ma T Y, Liu R S, et al. Toward fast, flexible, and robust low-light image enhancement [C]//2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022: 5627-5636.
[24] Hu J, Shen L, Sun G. Squeeze-and-excitation networks [C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018: 7132-7141.
[25] Jaderberg M, Simonyan K, Zisserman A, et al. Spatial transformer networks [DB/OL]. 2015[2023-11-15]. http://arxiv.org/abs/1506.02025.
[26] Woo S, Park J, Lee J Y, et al. CBAM: convolutional block attention module [C]//European Conference on Computer Vision, 2018: 3-19.
[27] Hu Y M, He H, Xu C X, et al. Exposure: a white-box photo post-processing framework [DB/OL]. 2017[2023-11-15]. http://arxiv.org/abs/1709.09602.
[28] Hwang S J, Kapoor A, Kang S B. Context-based automatic local image enhancement [C]//European Conference on Computer Vision, 2012: 569-582.
