面向视频的人脸特征计算方法

doi:10.3969/j.issn.0255-8297.2025.01.010

摘要/Abstract

摘要： 本文梳理了近五年视频人脸识别领域的研究成果，对比分析了采用的面向视频的人脸特征计算方法，主要分为传统人脸特征计算方法、深度学习人脸特征计算方法、特征聚合和特征融合方法。传统特征提取方法包括线性的和非线性的，深度学习特征提取方法包括非时空特征提取方法和时空特征提取方法。特征聚合和特征融合方法能够整合多个特征源以及融合不同时间段的特征，提高识别性能。此外，本文还统一分析了相关文献用到的算法、算法的优势、评价指标以及应用，能为实际应用中的视频人脸识别系统提供更可靠和高效的解决方案。

关键词: 视频人脸识别, 特征提取, 特征聚合, 特征融合, 深度学习

Abstract: This paper presents a review of research in video face recognition conducted over the past five years. It provides a comparative analysis of the facial feature computation methods, categorizing them into traditional approaches, deep learning techniques, and feature aggregation/fusion methods. Traditional feature extraction methods include linear and nonlinear approaches, while deep learning methods include spatial and temporal feature extraction techniques. Feature aggregation and fusion methods integrate multiple feature sources and fuse features from different time periods to improve recognition performance. At the end of each subsection, this paper also provides a unified analysis of the algorithms used in the literature, highlighting their advantages, evaluation metrics, and applications. Through this research, we aim to provide more reliable and efficient solutions for practical applications of video face recognition systems and promote further advancements in this field.

Key words: video face recognition, feature extraction, feature aggregation, feature fusion, deep learning

中图分类号:

TP39

王莹笑, 杨彦红, 谭云峰. 面向视频的人脸特征计算方法[J]. 应用科学学报, 2025, 43(1): 137-153.

WANG Yingxiao, YANG Yanhong, TAN Yunfeng. Video-Based Facial Feature Computation Methods[J]. Journal of Applied Sciences, 2025, 43(1): 137-153.

参考文献

[1] Omarov B, Omarov B, Shekerbekova S, et al. Applying face recognition in video surveillance security systems [C]//International Conference on Software Technology: Methods and Tools, 2019: 271-280.
[2] Suma V. Computer vision for human-machine interaction-review [J]. Journal of Trends in Computer Science and Smart technology, 2019(2): 131-139.
[3] Dharrao D S, Uke N J. Fractional krill-lion algorithm based actor critic neural network for face recognition in real time surveillance videos [J]. International Journal of Computational Intelligence and Applications, 2019, 18(2): 1950011.
[4] 朱国华, 徐昆. HOG的一种改进算法在人脸检测上的应用[J]. 计算机仿真, 2021, 38(9): 185-189. Zhu G H, Xu K. An improved HOG algorithm for face detection application [J]. Computer Simulation, 2021, 38(9): 185-189. (in Chinese)
[5] Akbay Y O, Mitrea M. Face detection and recognition in organic video: a comparative study for sport celebrities database [J]. Electronic Imaging, 2022, 34(10): 355-1-355-6.
[6] Li Q, Guo N, Ye X, et al. Video face recognition system: retinaface-mnet-faster and secondary search [C]//Advances in Information and Communication: Proceedings of the 2021 Future of Information and Communication Conference (FICC), 2021: 625-636.
[7] Abed R, Bahroun S, Zagrouba E. KeyFrame extraction based on face quality measurement and convolutional neural network for efficient face recognition in videos [J]. Multimedia Tools and Applications, 2021, 80(15): 23157-23179.
[8] 崔希, 刘首明. 基于朴素贝叶斯算法的智能电厂监控夜间视频人脸识别方法[J]. 电子测试, 2021(24): 44-46. Cui X, Liu S M. Face recognition method of intelligent power plant monitoring night video based on Naive Bayes algorithm [J]. Electronic Test, 2021(24): 44-46. (in Chinese)
[9] 吕婧淑, 操晓春, 杨培. 基于区块链和人脸识别的双因子身份认证模型[J]. 应用科学学报, 2019, 37(2): 164-178. Lyu J S, Cao X C, Yang P. Two-factor identity authentication model based on blockchain and face recognition [J]. Journal of Applied Sciences, 2019, 37(2): 164-178. (in Chinese)
[10] Tsai T H, Tsai C E, Chi P T. A one-shot face detection and recognition using deep learning method for access control system [J]. Signal, Image and Video Processing, 2023, 17(4): 1571- 1579.
[11] Yoganand A V, Kavida A C, Rukmanidevi D. An efficient PCA based pose and occlusion invariant face recognition system for video surveillance [J]. Cluster Computing, 2019, 22(5): 11443-11456.
[12] Li C, Wen C L, Qiu Y T. A video sequence face expression recognition method based on squeeze-and-excitation and 3DPCA network [J]. Sensors, 2023, 23(2): 823.
[13] Greche L, Akil M, Kachouri R, et al. A new pipeline for the recognition of universal expressions of multiple faces in a video sequence [J]. Journal of Real-Time Image Processing, 2020, 17(5): 1389-1402.
[14] Karpagam M, Jeyavathana R B, Chinnappan S K, et al. A novel face recognition model for fighting against human trafficking in surveillance videos and rescuing victims [J]. Soft Computing, 2023, 27(18): 13165-13180.
[15] Wang H, Zhang D S, Miao Z H. Face recognition with single sample per person using HOGLDB and SVDL [J]. Signal, Image and Video Processing, 2019, 13(5): 985-992.
[16] Lucey P, Cohn J F, Kanade T, et al. The extended Cohn-Kanade dataset (ck+): a complete dataset for action unit and emotion-specified expression [C]//2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2010: 94-101.
[17] Zhao W, Chellappa R, Phillips P J, et al. Face recognition: a literature survey [J]. ACM Computing Surveys (CSUR), 2003, 35(4): 399-458.
[18] 郑秋文, 刘惠义. 结合深度Q学习和注意模型的视频人脸识别[J]. 信息技术, 2019, 43(4): 111-115, 120. Zheng Q W, Liu H Y. Video face recognition combined with depth Q learning and attention model [J]. Information Technology, 2019, 43(4): 111-115, 120. (in Chinese)
[19] Moustafa A A, Elnakib A, Areed N F F. Age-invariant face recognition based on deep features analysis [J]. Signal, Image and Video Processing, 2020, 14(5): 1027-1034.
[20] Shan X X, Lu Y, Li Q L, et al. Model-based transfer learning and sparse coding for partial face recognition [J]. IEEE Transactions on Circuits and Systems for Video Technology, 2021, 31(11): 4347-4356.
[21] 何晓云, 许江淳, 史鹏坤, 等. 基于注意力机制的视频人脸表情识别[J]. 信息技术, 2020, 44(2): 103-107. He X Y, Xu J C, Shi P K, et al. Video facial expression recognition method based on attention mechanism [J]. Information Technology, 2020, 44(2): 103-107.(in Chinese)
[22] Hariri W. Efficient masked face recognition method during the COVID-19 pandemic [J]. Signal, Image and Video Processing, 2022, 16(3): 605-612.
[23] 晏鹏程, 张一鸣, 童光红, 等. 基于卷积神经网络的视频监控人脸识别方法[J]. 成都工业学院学报, 2020, 23(1): 26-31. Yan P C, Zhang Y M, Tong G H, et al. Face recognition method in video surveillance based on convolutional neural network [J]. Journal of Chengdu Technological University, 2020, 23(1): 26-31. (in Chinese)
[24] Huo J H, Van Zyl T L. Unique faces recognition in videos [C]//2020 IEEE 23rd International Conference on Information Fusion (FUSION), 2020: 1-7.
[25] Wolf L, Hassner T, Maoz I. Face recognition in unconstrained videos with matched background similarity [C]//Computer Vision & Pattern Recognit (CVPR), 2011: 529-534.
[26] Ricanek K, Tesafaye T. MORPH: a longitudinal image database of normal adult ageprogression [C]//7th International Conference on Automatic Face and Gesture Recognition (FGR06), 2006: 341-345.
[27] Lanitis A, Taylor C J, Cootes T F. Toward automatic simulation of aging effects on face images [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2002, 24(4): 442- 455.
[28] Huang G B, Mattar M A, Berg T L, et al. Labeled faces in the wild: a database for studying face recognition in unconstrained environments [J]. Month, 2008: 1-15.
[29] Martinez A M, Kak A C. PCA versus LDA [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2001, 23(2): 228-233.
[30] Dhall A, Goecke R, Lucey S, et al. Collecting large, richly annotated facial-expression databases from movies [J]. IEEE Multimedia, 2012, 19(3): 34-41.
[31] Wang Z Y, Huang B J, Wang G C, et al. Masked face recognition dataset and application [J]. IEEE Transactions on Biometrics, Behavior, and Identity Science, 2023, 5(2): 298-304.
[32] Liu Z X, Hu H, Bai J Q, et al. Feature aggregation network for video face recognition [C]//2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), 2019: 990-998.
[33] Peng B Y, Jin X, Wu Y C, et al. Geometry guided feature aggregation in video face recognition [C]//2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), 2019: 2670-2677.
[34] Protsenko I, Lehinevych T, Voitekh D, et al. Self-attention aggregation network for video face representation and recognition [DB/OL]. 2020[2024-07-17]. http://arxiv.org/abs/2010.05340v1.
[35] Gong S X, Shi Y C, Jain A K. Low quality video face recognition: multi-mode aggregation recurrent network (MARN) [C]//2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), 2019: 1027-1035.
[36] Gong S X, Shi Y C, Jain A K. Recurrent embedding aggregation network for video face recognition [DB/OL]. 2019[2024-07-17]. http://arxiv.org/abs/1904.12019v2.
[37] Gong S X, Shi Y C, Kalka N D, et al. Video face recognition: component-wise feature aggregation network (C-FAN) [C]//2019 International Conference on Biometrics (ICB), 2019: 1-8.
[38] Zhao H, Shi Y J, Tong X, et al. G-FAN: graph-based feature aggregation network for video face recognition [C]//2020 25th International Conference on Pattern Recognition (ICPR), 2021: 1672-1678.
[39] Zhang L Y, Wang H B, Wang H T. A feature map aggregation network for unconstrained video face recognition [J]. Journal of Intelligent & Fuzzy Systems, 2023, 44(2): 2413-2425.
[40] Hörmann S, Cao Z X, Knoche M, et al. Face aggregation network for video face recognition [C]//2021 IEEE International Conference on Image Processing (ICIP), 2021: 2973-2977.
[41] Wei J, Ying C. Aggregative adversarial network for still-to-video face recognition [C]//20205th International Conference on Computer and Communication Systems (ICCCS), 2020: 266-270.
[42] Ou Z H, Hu Y C, Song M N, et al. Redundancy removing aggregation network with distance calibration for video face recognition [J]. IEEE Internet of Things Journal, 2021, 8(9): 7279-7287.
[43] Rivero-Hernandez J, Morales-Gonzalez A, Denis L G, et al. Ordered weighted aggregation networks for video face recognition [J]. Pattern Recognition Letters, 2021, 146: 237-243.
[44] Li Q, Wan Q, Lee S H, et al. Video face recognition with audio-visual aggregation network [C]//28th International Conference on Neural Information Processing (ICONIP), 2021: 150-161.
[45] 刘宇琦, 赵宏伟, 王玉. 一种基于QPSO优化的流形学习的视频人脸识别算法[J]. 自动化学报, 2020, 46(2): 256-263. Liu Y Q, Zhao H W, Wang Y. Video face recognition method based on QPSO and manifold learning [J]. Acta Automatica Sinica, 2020, 46(2): 256-263. (in Chinese)
[46] Alqaralleh E, Afaneh A, Toygar Ö. Masked face recognition using frontal and profile faces with multiple fusion levels [J]. Signal, Image and Video Processing, 2023, 17(4): 1375-1382.
[47] Rodríguez A M, Geradts Z, Worring M, et al. Improved likelihood ratios for surveillance video face recognition with multimodal feature pairing [C]//202311th International Workshop on Biometrics and Forensics (IWBF), 2023: 3503-3607.
[48] Klare B F, Klein B, Taborsky E, et al. Pushing the frontiers of unconstrained face detection and recognition: IARPA Janus Benchmark A [C]//2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015: 1931-1939.
[49] Maze B, Adams J, Duncan J A, et al. IARPA Janus benchmark - C: face dataset and protocol [C]//2018 International Conference on Biometrics (ICB), 2018: 158-165.
[50] Kalka N D, Maze B, Duncan J A, et al. IJB-S: IARPA Janus surveillance video benchmark [C]//2018 IEEE 9th International Conference on Biometrics Theory, Applications and Systems (BTAS), 2018: 529-537.
[51] Beveridge J R, Phillips P J, Bolme D S, et al. The challenge of face recognition from digital point-and-shoot cameras [C]//2013 IEEE Sixth International Conference on Biometrics: Theory, Applications and Systems (BTAS), 2013: 1-8.
[52] Huang Z W, Shan S G, Wang R P, et al. A benchmark and comparative study of videobased face recognition on COX face database [J]. IEEE Transactions on Image Processing, 2015, 24(12): 5967-5981.