基于混合采样和SE_ResNet_SVM的不平衡多分类研究

doi:10.3969/j.issn.0255-8297.2024.06.009

摘要/Abstract

摘要： 针对结构化多分类算法中不平衡数据集类别分布不均导致分类难度增加的问题，本文提出了一种基于混合采样、压缩与激励（squeeze and excitation,SE）模块、改进深度残差网络和支持向量机（support vector machines,SVM）的网络模型SNSMRS (SMOTEENN-mixed residual networks-SVM network)。首先，通过合成少数过采样和编辑最近邻技术来改善数据分布；然后，构建融合SE模块与通过融合批次归一化和群组归一化的深度残差网络来提取特征；最后，通过SVM进行输出网络模型。其中，SE模块增强了模型对特征的区分能力，提升了模型的鲁棒性；基于融合归一化的残差网络受批次大小的影响较小，并且避免了传统神经网络梯度消失和精度退化等问题，增强了网络的稳定性与准确度； SVM可以根据特征向量在空间上的分布进行全部特征的分割，特征利用率高，提高了模型的分类精度。在7个不同规模和领域的非平衡公开数据集上进行了对比和消融实验，结果表明，本文所提的网络模型SNSMRS不仅优于其他深度学习模型，而且相对于未改良的ResNet，Macro-F1和G-mean值分别提升了约3%和4%，同时在4个数据集上的Macro-F1和G-mean值均超过了95%。

关键词: 不平衡多分类, 混合采样, 压缩与激励模块, 群组归一化, ResNet, 支持向量机

Abstract: A network model SNSMRS (SMOTEENN-mixed residual networks-SVM network) based on hybrid sampling, squeeze and excitation (SE) module, improved deep residual network and support vector machines (SVM) is proposed to address the problem of uneven class distribution of unbalanced data sets in traditional structured multiclassification algorithms, which leads to increased classification difficulty. Firstly, the data distribution is improved by synthesizing minority oversampling and editing nearest neighbors technique. Then the features are extracted by combining SE module and a deep residual network, improved with batch normalization and group normalization. Finally, the network model uses support vector machine (SVM) to output the classification results. The SE module enhances the model’s feature differentiation ability and robustness. The improvements to the ResNet, through fusion normalization, mitigate issues such as gradient vanishing and accuracy degradation, and ensure stability and accuracy regardless of batch_size. Additionally, SVM enhances the classification accuracy by effectively utilizing feature vectors in space to classify and extract features. Comparison and ablation experiments are conducted on seven unbalanced public datasets of various sizes and domains. The experimental results show that the proposed model, SNSMRS, not only outperforms other deep learning models, but also increases the values of Macro-F1 and G-mean by approximately 3% and 4%, respectively, compared with the original ResNet. Macro-F1 and G-mean values of SNSMRS exceed 95% on four of the datasets, demonstrating its superior performance.

Key words: unbalanced multi-classification, mixed sampling, squeeze and excitation (SE) module, group normalization, ResNet, support vector machines (SVM)

中图分类号:

TP183

矫桂娥, 翁铜铜, 张文俊. 基于混合采样和SE_ResNet_SVM的不平衡多分类研究[J]. 应用科学学报, 2024, 42(6): 1000-1015.

JIAO Guie, WENG Tongtong, ZHANG Wenjun. Unbalanced Multiclassification Study Based on Mixed Sampling and SE_ResNet_SVM[J]. Journal of Applied Sciences, 2024, 42(6): 1000-1015.

参考文献

[1] Sun A X, Lim E P, Liu Y. On strategies for imbalanced text classification using SVM: a comparative study [J]. Decision Support Systems, 2009, 48(1): 191-201.
[2] Shilaskar S, Ghatol A. Diagnosis system for imbalanced multi-minority medical dataset [J]. Soft Computing, 2019, 23(13): 4789-4799.
[3] Benchaji I, Douzi S, Elouahidi B. Using genetic algorithm to improve classification of imbalanced datasets for credit card fraud detection [C]//2018 2nd Cyber Security in Networking Conference (CSNet), 2018: 1-5.
[4] Chawla N V, Bowyer K W, Hall L O, et al. SMOTE: synthetic minority over-sampling technique [J]. Journal of Artificial Intelligence Research, 2002, 16: 321-357.
[5] Ha J, Lee J S. A new under-sampling method using genetic algorithm for imbalanced data classification [C]//Proceedings of the 10th International Conference on Ubiquitous Information Management and Communication, 2016: 1-6.
[6] Zhang Y, Liu Q. On IoT intrusion detection based on data augmentation for enhancing learning on unbalanced samples [J]. Future Generation Computer Systems, 2022, 133: 213-227.
[7] Batista G E, Bazzan A L C, Monard M C. Balancing training data for automated annotation of keywords: a case study [C]//II Brazilian Workshop on Bioinformatics, 2003: 10-18.
[8] Li X, Zhang L. Unbalanced data processing using deep sparse learning technique [J]. Future Generation Computer Systems, 2021, 125: 480-484.
[9] Sun Y M, Kamel M S, Wong A K C, et al. Cost-sensitive boosting for classification of imbalanced data [J]. Pattern Recognition, 2007, 40(12): 3358-3378.
[10] Dufrenois F. A one-class kernel fisher criterion for outlier detection [J]. IEEE Transactions on Neural Networks and Learning Systems, 2015, 26(5): 982-994.
[11] Wang Y X. An ensemble learning imbalanced data classification method based on sample combination optimization [J]. Journal of Physics: Conference Series, 2019, 1284(1): 12035.
[12] Joshi M V, Kumar V, Agarwal R C. Evaluating boosting algorithms to classify rare classes: comparison and improvements [C]//2001 IEEE International Conference on Data Mining, 2001: 257-264.
[13] Scholkopf B, Mika S, Burges C J C, et al. Input space versus feature space in kernel-based methods [J]. IEEE Transactions on Neural Networks, 1999, 10(5): 1000-1017.
[14] Deng X H, Xu Y B, Chen L C, et al. Dynamic clustering method for imbalanced learning based on AdaBoost [J]. The Journal of Supercomputing, 2020, 76(12): 9716-9738.
[15] Wang S T, Liu S Y, Zhang J K, et al. A new method of diesel fuel brands identification: SMOTE oversampling combined with XGBoost ensemble learning [J]. Fuel, 2020, 282: 118848.
[16] Pandey M, Xu Z R, Sholle E, et al. Extraction of radiographic findings from unstructured thoracoabdominal computed tomography reports using convolutional neural network based natural language processing [J]. PLoS One, 2020, 15(7): e0236827.
[17] 葛晓伟, 李凯霞, 程铭. 基于CNN-SVM的护理不良事件文本分类研究[J]. 计算机工程与科学, 2020, 42(1): 161-166. Ge X W, Li K X, Cheng M. Text classification of nursing adverse events based on CNN-SVM [J]. Computer Engineering & Science, 2020, 42(1): 161-166. (in Chinese)
[18] Ahlawat S, Choudhary A. Hybrid CNN-SVM classifier for handwritten digit recognition [J]. Procedia Computer Science, 2020, 167: 2554-2560.
[19] Zhang L G, Sheng Z, Li Y B, et al. Retraction note: image object detection and semantic segmentation based on convolutional neural network [J]. Neural Computing and Applications, 2023, 35(4): 3579.
[20] 暴雨轩, 芦天亮, 杜彦辉, 等. 基于i_ResNet34模型和数据增强的深度伪造视频检测方法[J]. 计算机科学, 2021, 48(7): 77-85. Bao Y X, Lu T L, Du Y H, et al. Deepfake videos detection method based on i_ResNet34 model and data augmentation [J]. Computer Science, 2021, 48(7): 77-85. (in Chinese)
[21] 万鹏, 赵竣威, 朱明, 等. 基于改进ResNet50模型的大宗淡水鱼种类识别方法[J]. 农业工程学报, 2021, 37(12): 159-168. Wan P, Zhao J W, Zhu M, et al. Freshwater fish species identification method based on improved ResNet50 model [J]. Transactions of the Chinese Society of Agricultural Engineering, 2021, 37(12): 159-168. (in Chinese)
[22] He K M, Zhang X Y, Ren S Q, et al. Deep residual learning for image recognition [C]//2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016: 770-778.
[23] Siddan G, Palraj P. Foetal neurodegenerative disease classification using improved deep ResNet classification based VGG-19 feature extraction network [J]. Multimedia Tools and Applications, 2022, 81(2): 2393-2408.
[24] 吕振虎, 许新征, 张芳艳. 基于挤压激励的轻量化注意力机制模块[J]. 计算机应用, 2022, 42(8): 2353-2360. Lyu Z H, Xu X Z, Zhang F Y. Lightweight attention mechanism module based on squeeze and excitation [J]. Journal of Computer Applications, 2022, 42(8): 2353-2360. (in Chinese)
[25] Hu J, Shen L, Albanie S, et al. Squeeze-and-excitation networks [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, 42(8): 2011-2023.
[26] Xu X L, Li W S, Duan Q L. Transfer learning and SE-ResNet152 networks-based for smallscale unbalanced fish species identification [J]. Computers and Electronics in Agriculture, 2021, 180: 105878.
[27] Batista G E A P A, Prati R C, Monard M C. A study of the behavior of several methods for balancing machine learning training data [J]. ACM SIGKDD Explorations Newsletter, 2004, 6(1): 20-29.
[28] Hawla N V, Bowyer K W, Hall L O, et al. Smote: synthetic minority over-sampling technique [J]. Journal of Artificial Intelligence Research, 2002, 16: 321-357.
[29] Wu Y X, He K M. Group normalization [M]//Computer Vision – ECCV 2018. Cham: Springer International Publishing, 2018: 3-19.
[30] Hendrycks D, Gimpel K. Gaussian error linear units (GELUs) [DB/OL]. 2016[2023-05-06]. http://arxiv.org/abs/1606.08415v5.
[31] Arik S ö, Pfister T. TabNet: attentive interpretable tabular learning [J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2021, 35(8): 6679-6687.
[32] Sokolova M, Lapalme G. A systematic analysis of performance measures for classification tasks [J]. Information Processing & Management, 2009, 45(4): 427-437.
[33] Kubat M, Holte R, Matwin S. Learning when negative examples abound [C]//European Conference on Machine Learning. Springer, 1997: 146-153.
[34] 李蒙蒙, 刘艺, 李庚松, 等. 不平衡多分类算法综述[J]. 计算机应用, 2022, 42(11): 3307-3321. Li M M, Liu Y, Li G S, et al. Survey on imbalanced multi-class classification algorithms [J]. Journal of Computer Applications, 2022, 42(11): 3307-3321. (in Chinese)