This paper proposes a tensor model for video rain streaks removal based on total variational regularization and L2,1 norm to solve rain streaks shielding. Firstly, the prior information of rain streaks component and the video background are preprocessed to obtain the corresponding regularization condition, so as to enhance the sparsity of each part and facilitate the separation of rain streaks. Then, considering the existence of irregular dynamic objects in the video, a total variational regularization term is introduced to suppress the variation of background intensity and alleviate the misjudgment of rain streaks. The alternating direction method of multipliers (ADMM) can be used to effectively solve the proposed tensor model, and carried out a large number of experiments on the synthetic and real datasets. Experimental results show that the proposed method under dynamic background can effectively remove the video rain streaks and retain more background details simultaneously. Compared with other relevant methods, the proposed method has great advantages in three comprehensive quantitative performance measures of peak signal-to-noise ratio (PSNR), structural similarity (SSIM) and residual error (RES).
[1] Zhu Q S, Shao L, Heng P A, et al. A novel rain detection and removal approach using guided filtering and formation modeling[C]//2013 IEEE International Conference on Robotics and Biomimetics, 2013:563-567.
[2] Wang C, Shen M M, Yao C. Rain streak removal by multi-frame-based anisotropic filtering[J]. Multimedia Tools and Applications, 2017, 76(2):2019-2038.
[3] Barnum P, Kanade T, Narasimhan S G. Spatio-temporal frequency analysis for removing rain and snow from videos[C]//International Workshop on Photometric Analysis For Computer Vision, 2007:1-8.
[4] Santhaseelan V, Asari V K. Utilizing local phase information to remove rain from video[J]. International Journal of Computer Vision, 2015, 112(1):71-89.
[5] Chen Z, Shen J H. A new algorithm of rain (snow) removal in video[J]. Journal of Multimedia, 2013, 8(2):168-174.
[6] Fadili M J, Starck J L, Bobin J, et al. Image decomposition and separation using sparse representations:an overview[J]. Proceedings of the IEEE, 2010, 98(6):983-994.
[7] Kang L W, Lin C W, Lin C T, et al. Self-learning-based rain streak removal for image/video[C]//2012 IEEE International Symposium on Circuits and Systems, 2012:1871-1874.
[8] Sun S H, Fan S P, Wang Y C F. Exploiting image structural similarity for single image rain removal[C]//2014 IEEE International Conference on Image Processing, 2014:4482-4486.
[9] Ramya C, Subha Rani S. Rain removal in image sequence using sparse coding[M]//Communications in Computer and Information Science. Berlin, Heidelberg:Springer Berlin Heidelberg, 2012:361-370.
[10] Zheng X H, Liao Y H, Guo W, et al. Single-image-based rain and snow removal using multiguided filter[M]//Neural Information Processing. Berlin, Heidelberg:Springer Berlin Heidelberg, 2013:258-265.
[11] Wang Y L, Chen C, Zhu S Y, et al. A framework of single-image deraining method based on analysis of rain characteristics[C]//2016 IEEE International Conference on Image Processing, 2016:4087-4091.
[12] Chen D Y, Chen C C, Kang L W. Visual depth guided color image rain streaks removal using sparse coding[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2014, 24(8):1430-1455.
[13] Zhou Y, Han Y S, Zhou P C. A method for removing raindrops from a single image[J]. Journal of Graphics, 2015, 36(3):438-443.
[14] Li Y, Tan R T, Guo X J, et al. Rain streak removal using layer priors[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition, 2016:2736-2744.
[15] Chen Y L, Hsu C T. A generalized low-rank appearance model for spatio-temporally correlated rain streaks[C]//2013 IEEE International Conference on Computer Vision, 2013:1968-1975.
[16] Kim J H, Sim J Y, Kim C S. Video deraining and desnowing using temporal correlation and low-rank matrix completion[J]. IEEE Transactions on Image Processing:a Publication of the IEEE Signal Processing Society, 2015, 24(9):2658-2670.
[17] Okatani T, Yoshida T, Deguchi K. Efficient algorithm for low-rank matrix factorization with missing components and performance comparison of latest algorithms[C]//2011 International Conference on Computer Vision, 2011:842-849.
[18] Jiang T X, Huang T Z, Zhao X L, et al. A novel tensor-based video rain streaks removal approach via utilizing discriminatively intrinsic priors[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition, 2017:2818-2827.
[19] Li M H, Xie Q, Zhao Q, et al. Video rain streak removal by multiscale convolutional sparse coding[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018:6644-6653.
[20] Sun Z Y, Xiong S W, Liu R W. Directional regularized tensor modeling for video rain streaks removal[DB/OL]. 2019[2020-10-14]. https://arxiv.org/abs/1902.07090.
[21] Wang Y T, Zhao X L, Jiang T X, et al. A total variation and group sparsity based tensor optimization model for video rain streak removal[J]. Signal Processing:Image Communication, 2019, 73:96-108.
[22] Yang W H, Tan R T, Feng J S, et al. Deep joint rain detection and removal from a single image[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition, 2017:1685-1694.
[23] Fu X Y, Huang J B, Ding X H, et al. Clearing the skies:a deep network architecture for single-image rain removal[J]. IEEE Transactions on Image Processing:a Publication of the IEEE Signal Processing Society, 2017, 26(6):2944-2956.
[24] Liu J Y, Yang W H, Yang S, et al. D3R-net:dynamic routing residue recurrent network for video rain removal[J]. IEEE Transactions on Image Processing, 2019, 28(2):699-712.
[25] Liu R S, Jiang Z Y, Ma L, et al. Deep layer prior optimization for single image rain streaks removal[C]//2018 IEEE International Conference on Acoustics, Speech and Signal Processing, 2018:1408-1412.
[26] Guo X J, Ma Y. Generalized Tensor Total Variation minimization for visual data recovery?[C]//2015 IEEE Conference on Computer Vision and Pattern Recognition, 2015:3603-3611.
[27] Jiang Y Q, Jin X, Wu Z Y. Video inpainting based on joint gradient and noise minimization[C]//Advances in Multimedia Information Processing-PCM 2016, 2016:407-417.
[28] Liu R W, Shi L, Huang W H, et al. Generalized total variation-based MRI Rician denoising model with spatially adaptive regularization parameters[J]. Magnetic Resonance Imaging, 2014, 32(6):702-720.
[29] Rudin L I, Osher S, Fatemi E. Nonlinear total variation based noise removal algorithms[J]. Physica D:Nonlinear Phenomena, 1992, 60(1/2/3/4):259-268.
[30] Chan S H, Khoshabeh R, Gibson K B, et al. An augmented Lagrangian method for total variation video restoration[J]. IEEE Transactions on Image Processing:a Publication of the IEEE Signal Processing Society, 2011, 20(11):3097-3111.
[31] Nikolova M. An algorithm for total variation minimization and applications[J]. Journal of Mathematical Imaging and Vision, 2004, 20(1/2):89-97.
[32] Cao X C, Yang L, Guo X J. Total variation regularized RPCA for irregularly moving object detection under dynamic background[J]. IEEE Transactions on Cybernetics, 2016, 46(4):1014-1027.
[33] Boyd S, Parikh N, Chu E, et al. Distributed optimization and statistical learning via the alternating direction method of multipliers[J]. Foundations and Trends in Machine Learning, 2010, 3(1):1-122.
[34] Deng W, Yin W T, Zhang Y. Group sparse optimization by alternating direction method[C]//SPIE Optical Engineering Applications. San Diego, California, 2013, 8858:242-256.
