在线评论的静态多模态情感分析

doi:10.3969/j.issn.0255-8297.2022.01.003

Abstract

Abstract: This paper proposes a static multi-modal sentiment classification model based on Pre-LN Transformer. This model firstly extracts semantic features from reviews using the encoder in Pre-LN Transformer structure, in which the multi-head self-attention mechanism allows the model to learn relevant emotional information in different subspaces. Then our model extracts the image features according to ResNet in the reviews. On the basis of feature level fusion, the visual attention mechanism guides the sentiment classification of text and realizes the static multimodal sentiment analysis of online reviews. Experimental results show that our model improves the performance by 1.34% and 1.10% in evaluation accuracy than BiGRU-mVGG and Trans-mVGG on Yelp datasets, which verifies the effectiveness and feasibility of the proposed model.

Key words: sentiment analysis, static multimodal, online reviews, visual aspect attention

CLC Number:

TP391

WANG Kaixin, XU Xiujuan, LIU Yu, ZHAO Zhehuan, ZHAO Xiaowei. Static Multimodal Sentiment Analysis of Online Reviews[J]. Journal of Applied Sciences, 2022, 40(1): 25-35.

References

[1] 张亚洲, 戎璐, 宋大为, 等. 多模态情感分析研究综述[J]. 模式识别与人工智能, 2020, 33(5):426-438. Zhang Y Z, Rong L, Song D W, et al. A survey on multimodal sentiment analysis[J]. Pattern Recognition and Artificial Intelligence, 2020, 33(5):426-438. (in Chinese)
[2] 潘家辉, 何志鹏, 李自娜, 等. 多模态情绪识别研究综述[J]. 智能系统学报, 2020, 84(4):7-19. Pan J H, He Z P, Li Z N, et al. A review of multimodal emotion recognition[J]. CAAI Transactions on Intelligent Systems, 2020, 84(4):7-19. (in Chinese)
[3] Zadeh A, Chen M, Poria S, et al. Tensor fusion network for multimodal sentiment analysis[C]//Proceedings of 2017 Conference on Empirical Methods in Natural Language Processing, Copenhagen, Denmark, 2017:1103-1114.
[4] Chen M H, Wang S, Liang P P, et al. Multimodal sentiment analysis with word-level fusion and reinforcement learning[C]//Proceedings of the 19th ACM International Conference on Multimodal Interaction, Glasgow, UK, 2017:163-171.
[5] Cao D, Ji R, Lin D, et al. A cross-media public sentiment analysis system for microblog[J]. Multimedia Systems, 2016, 22(4):479-486.
[6] Yu Y, Lin H, Meng J, et al. Visual and textual sentiment analysis of a microblog using deep convolutional neural networks[J]. Algorithms, 2016, 9(2):41-52.
[7] Li Z H, Fan Y Y, Liu W H, et al. Image sentiment prediction based on textual descriptions with adjective noun pairs[J]. Multimedia Tools and Application, 2018, 77(1):1115-1132.
[8] 蔡国永, 夏彬彬. 基于卷积神经网络的图文融合媒体情感预测[J]. 计算机应用, 2016, 36(2):428-431. Cai G Y, Xia B B. Multimedia sentiment analysis based on convolutional neural network[J]. Journal of Computer Applications, 2016, 36(2):428-431. (in Chinese)
[9] Xu N, Mao W J. MultiSentiNet:a deep semantic network for multimodal sentiment analysis[C]//Proceedings of 2017 ACM on Conference on Information and Knowledge Management, Singapore, 2017:2399-2402.
[10] Truong T Q, Lauw H W. VistaNet:visual aspect attention network for multimodal sentiment analysis[C]//Proceedings of AAAI Conference on Artificial Intelligence, Hawaii, USA, 2019:305-312.
[11] Huang F, Zhang X, Zhao Z, et al. Image-text sentiment analysis via deep multimodal attentive fusion[J]. Knowledge-Based Systems, 2019, 167:26-37.
[12] 林敏鸿, 蒙祖强. 基于注意力神经网络的多模态情感分析[J]. 计算机科学, 2020, 47(增刊2):518-524, 558. Lin M H, Meng Z Q. Multimodal sentiment analysis based on attention neural network[J]. Computer Science, 2020, 47(Suppl. 2):518-524, 558. (in Chinese)
[13] Tang G, Müller M, Rios A, et al. Why self-attention? a targeted evaluation of neural machine translation architectures[C]//Proceedings of 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, 2018:4263-4272.
[14] Vaswani A, Bengio S, Brevdo E, et al. Tensor2Tensor for neural machine translation[C]//Proceedings of the 13th Conference of the Association for Machine Translation in Americas, Boston, USA, 2018:193-199.
[15] Wang Q, Li B, Xiao T, et al. Learning deep transformer models for machine translation[C]//Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, 2019:1810-1822.
[16] Xiong R, Yang Y, He D, et al. On layer normalization in the transformer architecture[C]//Proceedings of the Thirty-Seventh International Conference on Machine Learning, Virtual, 2020:10524-10533.
[17] Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition[C]//Proceedings of the 3rd International Conference on Learning Representations, San Diego, USA, 2015:1-14.
[18] Pennington J, Socher R, Manning C D. Glove:global vectors for word representation[C]//Proceedings of 2014 Conference on Empirical Methods in Natural Language Processing, Doha, Qatar, 2014:1532-1543.
[19] He K, Zhang X, Ren S, et al. Deep residual learning for image recognition[C]//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, USA, 2016:770-778.

Static Multimodal Sentiment Analysis of Online Reviews

PDF

Knowledge

Abstract

Cite this article

share this article

References

Related Articles 15

Recommended Articles

Metrics

Comments

[1]	ZHOU Chuanhua, XU Wenqian, ZHU Junjie. Ensemble Classification Algorithm Based on Cost Sensitive Convolutional Neural Networks [J]. Journal of Applied Sciences, 2022, 40(1): 69-79.
[2]	WEI Mingjun, ZHOU Taiyu, JI Zhanlin, ZHANG Xinnan. Mask Wearing Detection in Complex Scenes Based on Mask-YOLO [J]. Journal of Applied Sciences, 2022, 40(1): 93-104.
[3]	LEI Qianhui, PAN Lili, SHAO Weizhi, HU Haipeng, HUANG Yao. Segmentation Model of COVID-19 Lesions Based on Triple Attention Mechanism [J]. Journal of Applied Sciences, 2022, 40(1): 105-115.
[4]	WANG Peng, ZHENG Wenfeng, SHI Jin, JIN Shuo, LIU Zihao. Remote Sensing Image Object Detection Based on MFANet and Contextual Features Fusion [J]. Journal of Applied Sciences, 2022, 40(1): 131-144.
[5]	JI Deqiang, WANG Hairong, CHE Miao, WANG Jiaxin. KNN-GWD Recommendation Model and Its Application [J]. Journal of Applied Sciences, 2022, 40(1): 145-154.
[6]	LIU Xinghong, WANG Ying, WANG Xin, LAN Shumei. Heterogeneous Information Network Representation Learning Based on Generative Adversarial Network [J]. Journal of Applied Sciences, 2021, 39(4): 532-544.
[7]	JIANG Wenxuan, DUAN Youxiang, SUN Qifeng. Hybrid Feature Selection Algorithm Based on Mutual Information [J]. Journal of Applied Sciences, 2021, 39(4): 545-558.
[8]	ZHENG Changliang, PANG Ming. Spatial-Temporal Weight Attitude Motion Feature Extraction Algorithm Using Convolutional Neural Network [J]. Journal of Applied Sciences, 2021, 39(4): 594-604.
[9]	ZHANG Xiaolong, WANG Qingwei, LI Shangbin. Recognition Method of Human Dangerous Behavior in Multimodal Scenes Using Reinforcement Learning [J]. Journal of Applied Sciences, 2021, 39(4): 605-614.
[10]	LAI Yibin, LU Shenglian, QIAN Tingting, SONG Zhen, CHEN Ming. Three-Dimensional Point Cloud Segmentation for Plants [J]. Journal of Applied Sciences, 2021, 39(4): 660-671.
[11]	GUO Yubo, LU Jun, DUAN Pengqi. Automatic Classification of Bamboo Flute Playing Skills Based on Deep Learning [J]. Journal of Applied Sciences, 2021, 39(4): 685-694.
[12]	HAO Yan, SHI Huiyu, HUO Shoujun, HAN Dan, CAO Rui. Emotion Classification Based on EEG Deep Learning [J]. Journal of Applied Sciences, 2021, 39(3): 347-346.
[13]	SIMA Yi, YI Jizheng, CHEN Aibin, ZHOU Mengna. Fully Expression Frame Localization and Recognition Based on Dynamic Face Image Sequences [J]. Journal of Applied Sciences, 2021, 39(3): 357-356.
[14]	DU Chengze, DUAN Youxiang, SUN Qifeng. Seismic Fault Identification Method Based on ResUNet and Dense CRF Model [J]. Journal of Applied Sciences, 2021, 39(3): 367-366.
[15]	YU Qun, ZHANG Jianxin, WEI Xiaopeng, ZHANG Qiang. Cascaded Separable and Dilated Residual U-Net for Liver Tumor Segmentation [J]. Journal of Applied Sciences, 2021, 39(3): 378-377.