融合音频、文本、表情动作的多模态情感识别

doi:10.3969/j.issn.0255-8297.2023.01.005

Abstract

Abstract: For the problems of low accuracy and weak generalization ability in the process of human emotion recognition, a fusion method of multi-modal emotion recognition based on speech, text and motion is proposed. In the speech mode, a depth wavefield extrapolation-improved wave physics model (DWE-WPM) is designed to simulate the sequence information mining process of long short-term memory (LSTM) network; In the text mode, a transformer model with multi-attention mechanism is used to capture the potential semantic expression of emotion; In the motion mode, sequential features of facial expression and hand action are combined by using two-way three-layer LSTM model with attention mechanism. Accordingly, a multi-modal fusion scheme is designed to achieve high-precision and strong generalization ability of emotion recognition. In the general emotion corpus IEMOCAP, the method proposed in this paper is compared with existing emotion recognition algorithms. Experimental results show that the proposed method has higher recognition accuracy both in single modality and multi-modals, with average accuracy improved by 16.4% and 10.5% respectively, effectively improving the ability of human emotion recognition in human-computer interaction.

Key words: speech emotion recognition, text emotion recognition, motion emotion recognition, Transformer model, attention mechanism

CLC Number:

P751.1

JIA Ning, ZHENG Chunjun. Multi-modal Emotion Recognition Using Speech, Text and Motion[J]. Journal of Applied Sciences, 2023, 41(1): 55-70.

References

[1] Tiwari U, Soni M, Chakraborty R, et al. Multi-conditioning and data augmentation using generative noise model for speech emotion recognition in noisy conditions[C]//2020 IEEE International Conference on Acoustics, Speech and Signal Processing, 2020:7194-7198.
[2] Jermsittiparsert K, Abdurrahman A, Siriattakul P, et al. Pattern recognition and features selection for speech emotion recognition model using deep learning[J]. International Journal of Speech Technology, 2020, 23(4):1-8.
[3] 曾润华, 张树群. 改进卷积神经网络的语音情感识别方法[J]. 应用科学学报, 2018, 36(5):837-844. Zeng R H, Zhang S Q. Speech and emotional recognition method based on improving convolutional neural networks[J]. Journal of Applied Sciences, 2018, 36(5):837-844. (in Chinese)
[4] Chu Y, Li T G, Ye S, et al. Research on feature selection method in speech emotion recognition[J]. Journal of Applied Acoustics, 2020, 39(2):223-230.
[5] Wang W, Yang L P, Wei L. Extraction and analysis of speech emotion characteristics[J]. Research and Exploration in Laboratory, 2013, 32(7):91-94.
[6] Yang M H, Tao J H, Li H, et al. Nature multimodal human-computer-interaction dialog system[J]. Computer Science, 2014, 41(10):12-18.
[7] Hughes T W, Williamson I A D, Minkov M, et al. Wave physics as an analog recurrent neural network[J]. Science Advances, 2019, 5(12):6946-6958.
[8] Bouazizi M, Ohtsuki T. Multi-class sentiment analysis on Twitter:classification performance and challenges[J]. Big Data Mining and Analytics, 2019, 3:181-194.
[9] Liang Y, Meng F, Zhang J, et al. A dependency syntactic knowledge augmented interactive architecture for end-to-end aspect-based sentiment analysis[J]. Neurocomputing, 2020, 454:291-302.
[10] 司马懿, 易积政, 陈爱斌, 等. 动态人脸图像序列中表情完全帧的定位与识别[J]. 应用科学学报, 2021, 39(3):357-366. Si M Y, Yi J Z, Chen A B, et al. Fully expression frame localization and recognition based on dynamic face image sequences[J]. Journal of Applied Sciences, 2021, 39(3):357-366. (in Chinese)
[11] Jain D K, Shamsolmoali P, Sehdev P. Extended deep neural network for facial emotion recognition[J]. Pattern Recognition Letters, 2019, 120:69-74.
[12] Thomas K, Pranav E, Supriya M H. A generalized deep learning model for denoising image datasets[J]. International Journal of Engineering and Advanced Technology, 2020, 10(1):9-14.
[13] Ly S T, Lee G S, Kim S H, et al. Gesture-based emotion recognition by 3D-CNN and LSTM with keyframes selection[J]. International Journal of Contents, 2019, 15(4):59-64.
[14] Busso C, Bulut M, Lee C C, et al. IEMOCAP:interactive emotional dyadic motion capture database[J]. Language Resources and Evaluation, 2008, 42(4):335-359.
[15] Poria S, Majumder N, Hazarika D, et al. Multimodal sentiment analysis:addressing key issues and setting up the baselines[J]. IEEE Intelligent Systems, 2018, 33(6):17-25.
[16] Sahu G. Multimodal speech emotion recognition and ambiguity resolution[EB/OL]. (2019-04-12)[2021-08-21]. https://arxiv.org/abs/1904.06022v1.
[17] Happy S L, Dantcheva A, Bremond F, et al. Expression recognition with deep features extracted from holistic and part-based models[J]. Image and Vision Computing, 2021, 105(1):104038.1-104038.11.
[18] Tripathi S, Beigi H. Multi-modal emotion recognition on IEMOCAP dataset using deep learning[EB/OL]. (2019-11-06)[2021-09-04]. https://arxiv.org/abs/1804.05788v3.
[19] Ren M, Nie W, Liu A, et al. Multi-modal correlated network for emotion recognition in speech[J]. Visual Informatics, 2019, 3(3):150-155.
[20] Mirsamadi S, Barsoum E, Zhang C. Automatic speech emotion recognition using recurrent neural networks with local attention[C]//2017 IEEE International Conference on Acoustics, Speech and Signal Processing, 2017:2227-2231.
[21] Chen M, Zhao X D. A multi-scale fusion framework for bimodal speech emotion recognition[C]//Interspeech 2020, 2020:374-378.

Multi-modal Emotion Recognition Using Speech, Text and Motion

PDF

Knowledge

Abstract

Cite this article

share this article

References

Related Articles 14

Recommended Articles

Metrics

Comments

[1]	XIONG Juan, ZHANG Sunjie, KAN Yaya, CHEN Jiahao. Remote Sensing Image Object Detection Based on CAFPN and Refinement Double-Head Decoupling [J]. Journal of Applied Sciences, 2023, 41(6): 989-1003.
[2]	HAN Huiyan, WU Weizhou, WANG Wenjun, HAN Xie. Research on Pipe Sorting Based on Improved Mask R-CNN Using Feature Fusion and Region Generation Network [J]. Journal of Applied Sciences, 2023, 41(5): 840-854.
[3]	ZHANG Chunsen, ZHU Jiangle, ZHANG Xuefen, LIU Xudong, SHI Shu. Void Filling of DEM in a Generative Adversarial Network Fused with Self-Attention Mechanism [J]. Journal of Applied Sciences, 2023, 41(5): 789-800.
[4]	LIU Fei, LI Xin. Cloud Detection of Landsat Images Based on Attention U-Net [J]. Journal of Applied Sciences, 2022, 40(6): 906-917.
[5]	CHEN Lifang, WEI Mengru. 3D Point Cloud Classification and Segmentation Network Based on Local Feature Enhancement [J]. Journal of Applied Sciences, 2022, 40(2): 328-337.
[6]	LEI Qianhui, PAN Lili, SHAO Weizhi, HU Haipeng, HUANG Yao. Segmentation Model of COVID-19 Lesions Based on Triple Attention Mechanism [J]. Journal of Applied Sciences, 2022, 40(1): 105-115.
[7]	WEI Mingjun, ZHOU Taiyu, JI Zhanlin, ZHANG Xinnan. Mask Wearing Detection in Complex Scenes Based on Mask-YOLO [J]. Journal of Applied Sciences, 2022, 40(1): 93-104.
[8]	FAN Shouxiang, YAO Junping, LI Xiaojun, CHENG Kaiyuan. A Method for Aspect-Level Sentiment Classification Based on Multi-pattern Feature Fusion [J]. Journal of Applied Sciences, 2021, 39(6): 969-982.
[9]	WANG Shengke, REN Pengfei, Lü Xin, ZHUANG Xinfa. Small Target Detection Algorithm of UAV High Resolution Image Based on Center Point and Dual Attention Mechanism [J]. Journal of Applied Sciences, 2021, 39(4): 650-659.
[10]	PENG Ning, CHEN Aibin, ZHOU Guoxiong, CHEN Wenjie, LIU Jing. Environmental Sound Recognition Based on Attention Sinusoidal Representation Network [J]. Journal of Applied Sciences, 2021, 39(4): 641-649.
[11]	ZHAO Yunshan, DUAN Youxiang. Convolutional Neural Networks Text Classification Model Based on Attention Mechanism [J]. Journal of Applied Sciences, 2019, 37(4): 541-550.
[12]	JIN Huazhong, LIU Xiaolong, HU Zike. An Image Caption Generation Model Combining Global and Local Features [J]. Journal of Applied Sciences, 2019, 37(4): 501-509.
[13]	ZENG Run-hua, ZHANG Shu-qun. Speech and Emotional Recognition Method Based on Improving Convolutional Neural Networks [J]. Journal of Applied Sciences, 2018, 36(5): 837-844.
[14]	JIA Qi, Lü Xu-liang, WU Chao, TANG Hou-chao. Evaluation of Camouflage Effectiveness Using Human Visual Attention Mechanism [J]. Journal of Applied Sciences, 2011, 29(3): 294-298.