Short Text Sentiment Analysis Based on Self-attention and Capsule Network

doi:10.3969/j.issn.1006-2475.2020.07.012

Abstract

Abstract: Sentiment analysis of short texts is a challenging task. Aiming at the shortcomings of traditional convolutional neural networks and recurrent neural networks that can not fully obtain the semantic information contained in texts, this paper proposed a model that used the multi-head self-attention layer as the feature extractor and used the capsule network as the classification layer. The model can extract rich text information and has strong expressive ability. Experimental results on Chinese text showed that compared with the traditional deep learning method, the proposed model improved the accuracy of sentiment analysis. In the small dataset and cross-domain migration, compared with traditional method, the accuracy was greatly improved.

Key words: emotional analysis, self-attention mechanism, capsule network, small datasets learning, migration learning

CLC Number:

TP311

XU Long. Short Text Sentiment Analysis Based on Self-attention and Capsule Network[J]. Computer and Modernization, 2020, 0(07): 61-64.

References

［1］ JIN H W, HUANG M L, ZHU X Y. Sentiment analysis with multi-source product reviews［C］// International Conference on Intelligent Computing. Springer Berlin Heidelberg, 2012:301-308.
［2］陆文星,王燕飞. 中文文本情感分析研究综述［J］. 计算机应用研究, 2012,29(6):2014-2017.
［3］ PANG B, LEE L. A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts［C］// Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics. ACM, 2004:DOI:10.3115/1218955.1218990.
［4］ PANG B， LEE L, VAITHYANATHAN S. Thumbs up?Sentiment classification using machine learning techniques［C］// 〖JP2〗Proceedings of the Conference on Empirical Methods in Natural Language Processing(EMNLP). 2002:79-86.
［5］ MOHAMMAD S M, KIRITCHENKO S, ZHU X. NRC-Canada: Building the state-of-the-art in sentiment analysis of tweets［J］. arXiv preprint arXiv:1308.6242, 2013.
［6］何炎祥,孙松涛,牛菲菲,等. 用于微博情感分析的一种情感语义增强的深度学习模型［J］. 计算机学报, 2017,40(4):773-790.
［7］ VO Q H, NGUYEN H T, LE B, et al. Multi-channel LSTM-CNN model for Vietnamese sentiment analysis［C］// 2017 the 9th International Conference on Knowledge and Systems Engineering (KSE). IEEE, 2017:DOI:10.1109/KSE.2017.8119429.

［8］ HAO Y Z, ZHENG Q H, LAN Y Y, et al. Improving Chinese sentiment analysis via segmentation-based representation using parallel CNN［C］// International Conference on Advanced Data Mining and Applications. Springer, 2017:668-680.

［9］ KIM Y. Convolutional neural networks for sentence classification［J］. arXiv preprint arXiv:1408.5882, 2014.
［10］CAI G Y, XIA B B. Convolutional neural networks for multimedia sentiment analysis［C］// CCF International Conference on Natural Language Processing and Chinese Computing. Springer, 2015:159-167.
［11］SEVERYN A, MOSCHITTI A. Twitter sentiment analysis with deep convolutional neural networks［C］// Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 2015:959-962.
［12］BAGHERI H, ISLAM M J. Sentiment analysis of twitter data［J］. arXiv preprint arXiv:1711.10377, 2017.
［13］LIN Z H, FENG M W, SANTOS C N, et al. A structured self-attentive sentence embedding［J］. arXiv preprint arXiv:1703.03130, 2017.
［14］PAPPAS N, POPESCU-BELIS A. Multilingual hierarchical attention networks for document classification［J］. arXiv preprint arXiv:1707.00896, 2017.
［15］RADFORD A, NARASIMHAN K, SALIMANS T, et al. Improving Language Understanding with Unsupervised Learning［R］. Technical Report, OpenAI, 2018.
［16］DEVLIN J, CHANG M W, LEE K, et al. Bert: Pre-training of deep bidirectional transformers for language understanding［J］. arXiv preprint arXiv:1810.04805, 2018.
［17］SABOUR S, FROSST N, HINTON G E. Dynamic routing between capsules［C］// Advances in Neural Information Processing Systems. 2017:3856-3866.
［18］HINTON G E, SABOUR S, FROSST N. Matrix capsules with EM routing［C］// The 6th International Conference on Learning Representations(ICLR). 2018:789.
［19］张越,夏鸿斌. 基于弱监督预训练CNN模型的情感分析方法［J］. 计算机工程与应用, 2018,908(13):33-39.
［20］MIKOLOV T, CHEN K, CORRADO G, et al. Efficient estimation of word representations in vector space［J］. arXiv preprint arXiv:1301.3781, 2013.

[1]	ZHANG Xiaodong1, BAI Guangzhi1, LI Min1, LI Haoyang2. Oil and Gas Well Production Prediction Model Based on Empirical Wavelet Transform [J]. Computer and Modernization, 2024, 0(12): 53-58.
[2]	HE Sida, CHEN Pinghua. Intent-based Lightweight Self-Attention Network for Sequential Recommendation [J]. Computer and Modernization, 2024, 0(12): 1-9.
[3]	QI Xian, LIU Daming, CHANG Jiaxin. Multi-view 3D Reconstruction Based on Improved Self-attention Mechanism [J]. Computer and Modernization, 2024, 0(11): 106-112.
[4]	HOU Congying, YANG Wengqing, WANG Zhao, CHENG Cong. Speech Enhancement Based on Time-frequency Self-attention Residual Temporal#br# Convolutional Networks [J]. Computer and Modernization, 2024, 0(09): 20-24.
[5]	ZHANG Huinan1, ZHANG Qiang1, SUN Hongxia2. Dynamic Analysis Model of Reservoir Production Based on Improved#br# Time-series Capsule Network [J]. Computer and Modernization, 2024, 0(09): 15-19.
[6]	YE Senhui, WANG Lei. Multi-view Reconstruction with Local Self-attention and Deep Optimization [J]. Computer and Modernization, 2024, 0(05): 92-98.
[7]	YOU Jiajing1, 2, HE Yueshun1, HE Linlin1, ZHONG Hailong1, 2. Encryption Traffic Classification Method Based on AHP-CNN [J]. Computer and Modernization, 2024, 0(04): 83-87.
[8]	XIA Yi-chun, LI Wang-gen, LI Dou-dou, GE Ying-kui, WANG Zhi-ge. CTR Prediction Model Combining Attention Mechanism and Graph Neural Network [J]. Computer and Modernization, 2023, 0(03): 29-37.
[9]	LI Wei-song, TANG Min-fang, HE Zheng-ling, WANG Peng, DU Li-dong, FANG Zhen, CHEN Xian-xiang. Automatic Sleep Staging Algorithm Based on Self-attention Mechanism and Single Lead ECG [J]. Computer and Modernization, 2022, 0(12): 50-59.
[10]	WAN Fa-yang, YU Xu, XU Qi-jiang. Deep Bug Triage Model Based on Multi-head Self-attention Mechanism [J]. Computer and Modernization, 2021, 0(11): 39-43.
[11]	SONG Shuang, LU Xin-da. Text Matching Model Based on BERT and Self-attention Mechanism of Image [J]. Computer and Modernization, 2021, 0(11): 12-16.
[12]	MO Bei-bei, WU Ke-he . Research on Intelligent Detection Technology for Illegal Wearing #br# in Power Operation by Introducing Self-Attention [J]. Computer and Modernization, 2020, 0(02): 115-.