News Label Classification Based on BERT and Deep Equal Length Convolution

Abstract

Abstract: For the THUCNews’ Chinese news text label classification task, a news label classification model (DPCNN-BERT) that combines multi-layer equal-length convolution and residual connection based on BERT pre-training language model is proposed. Firstly, by querying the Chinese vector table, each word in the news text is converted into a vector and input into BERT model to get the full-text context of the text. Then, the local context relationship in the text is obtained through the initial semantic extraction layer and deep equal-length convolution. Finally, the predicted label of the entire news text is obtained through a single-layer fully connected neural network. The model proposed in this paper is compared with the convolutional Neural Network Classification Model (TextCNN), Recurrent Neural Network Classification Model (TextRNN) and other models. The experimental results show that the prediction accuracy of the model reaches 94.68%, and the F1 value reaches 94.67%, which is better than the comparison models. The performance of the model proposed in this paper is verified.

Key words: label classification, equal-length convolution, residual connection, BERT

YANG Wen-hao, LIU Guang-cong, LUO Ke-jing. News Label Classification Based on BERT and Deep Equal Length Convolution [J]. Computer and Modernization, 2021, 0(08): 94-99.

References ［23］

［1］	汪岿,刘柏嵩. 文本分类研究综述［J］. 数据通信, 2019(3):37-47.
［2］	HUQ M R, ALI A, RAHMAN A. Sentiment analysis on Twitter data using KNN and SVM［J］. International Journal of Advanced Computer Science and Applications, 2017,8(6): 19-25.
［3］	RONG X. Word2vec parameter learning explained［J］. arXiv preprint arXiv:1411.2738, 2014.
［4］	何力,谭霜,项凤涛,等. 基于深度学习的文本分类技术研究进展［J/OL］.计算机工程:1-15［2020-11-22］. https://doi.org/10.19678/j.issn.1000-3428.0059099.
［5］	NAM J, KIM J, MENCIA E L, et al. Large-scale multi-label text classification—revisiting neural networks［C］// 2014 European Conference on Machine Learning and Knowledge Discovery in Databases. 2014:437-452.
［6］	BENGIO Y, DUCHARME R, VINCENT P, et al. A neural probabilistic language model［J］. Journal of Machine Learning Research, 2003,3:1137-1155.
［7］	KIM Y. Convolutional neural networks for sentence classification［J］. arXiv preprint arXiv:1408.5882, 2014.
［8］	BAGHERI H, ISLAM M J. Sentiment analysis of Twitter data［J］. arXiv preprint arXiv:1711.10377, 2017.
［9］	ZHANG Y, WALLACE B. A sensitivity analysis of (and practitioners’ guide to) convolutional neural networks for sentence classification［J］. arXiv preprint arXiv:1510.03820, 2015.
［10］	PETERS M E, NEUMANN M, IYYER M, et al. Deep contextualized word representations［J］. arXiv preprint arXiv:1802.05365, 2018.
［11］	DEVLIN J, CHANG M W, LEE K, et al. Bert: Pretraining of deep bidirectional transformers for language understanding［J］. arXiv preprint arXiv:1810.04805, 2018.
［12］	LUONG M T, PHAM H, MANNING C D. Effective approaches to attention-based neural machine translation［J］. arXiv preprint arXiv:1508.04025, 2015.
［13］	BAHDANAU D, CHO K, BENGIO Y. Neural machine translation by jointly learning to align and translate［J］. arXiv preprint arXiv:1409.0473, 2014.
［14］	VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need［C］// Proceedings of the 31st International Conference on Neural Information Processing. 2017: 6000-6010.
［15］	HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition［C］// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016:770-778.
［16］	IOFFE S，SZEGEDY C. Batch normalization: Accelerating deep network training by reducing internal covariate shift ［C］// 2015 International Conference on Machine Learning. 2015:448-456.
［17］	SANTURKAR S, TSIPRAS D, ILYAS A, et al. How does batch normalization help optimization?［C］//Advances in Neural Information Processing Systems. 2018:2483-2493.
［18］	SCHOLZ R W, TIETJE O. Embedded Case Study Methods: Integrating Quantitative and Qualitative Knowledge［M］. Sage, 2002.
［19］	ZHANG Z Y, HAN X, LIU Z Y, et al. ERNIE: Enhanced language representation with informative entities［J］. arXiv preprint arXiv:1905.07129, 2019.
［20］	LAN Z, CHEN M, GOODMAN S, et al. Albert: A lite BERT for self-supervised learning of language representations［J］. arXiv preprint arXiv:1909.11942, 2019.
［21］	YANG Z C, YANG D Y, DYER C, et al. Hierarchical attention networks for document classification［C］// Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2016:1480-1489.
［22］	CHO K, VAN MERRIENBOER B, GULCEHRE C, et al. Learning phrase representations using RNN encoder-decoder for statistical machine translation［J］. arXiv preprint arXiv:1406.1078, 2014.
［23］	方炯焜,陈平华,廖文雄. 结合GloVe和GRU的文本分类模型［J］. 计算机工程与应用, 2020,56(20):98-103.

[1]	ZHENG Li-rui, XIAO Xiao-xia, ZOU Bei-ji, LIU Bin, ZHOU Zhan. Named Entity Recognition in Electronic Medical Record Based on BERT [J]. Computer and Modernization, 2024, 0(01): 87-91.
[2]	LIU Yu-peng, GE Yan, DU Jun-wei, CHEN Zhuo. Joint Extraction Method of Entities and Relations Based on FGM and Pointer Annotation [J]. Computer and Modernization, 2023, 0(11): 1-5.
[3]	TANG Shi-qi, ZHOU Rui-ping, XIE Shi-bin, LIU Meng-chi, XIAO Wen, . Cross-language Multi-label Sentiment Classification Based on Stacked Denoising AutoEncoder [J]. Computer and Modernization, 2023, 0(11): 6-12.
[4]	LI Shi-yue, MENG Jia-na, YU Yu-hai, LI Xue-ying, XU Ying-ao. Aspect Based Sentiment Analysis Model Based on Knowledge Enhancement [J]. Computer and Modernization, 2023, 0(10): 1-8.
[5]	WANG Hong-jie, XU Sheng-chao. Clustering Method of Cloud Platform Abnormal Transmission Data Based on Hilbert Similarity [J]. Computer and Modernization, 2023, 0(09): 27-31.
[6]	ZENG Li-li, TANG Hua-bei, NIU Yi-xiao, MENG Fan-yue. Lithofacies Identification Method Based on LSTM Stacked Residual Network [J]. Computer and Modernization, 2023, 0(08): 38-43.
[7]	XIE Shi-chao, HUANG Wei, REN Xiang-hui. A Text Entity Linking Method Based on BERT [J]. Computer and Modernization, 2023, 0(02): 58-61.
[8]	ZHU Ya-jun, Yong Tso, Nyima Tashi, . Tibetan Medical Entity Recognition Based on Tibetan BERT [J]. Computer and Modernization, 2023, 0(01): 43-48.
[9]	YU Qing, MA Zhi-long, XU Chun. Medical Knowledge Extraction Based on BERT and Non-autoregressive [J]. Computer and Modernization, 2023, 0(01): 120-126.
[10]	HUANG Zhong-xiang, LI Ming. Text Classification Based on ALBERT Combined with Bidirectional Network [J]. Computer and Modernization, 2022, 0(10): 8-12.
[11]	ZHNAG Jun, QIU Long-long. A Text Classification Model Based on BERT and Pooling Operation [J]. Computer and Modernization, 2022, 0(06): 1-7.
[12]	CHEN Gang. Government Hotline Work-order Classification Fusing RoBERTa and Feature Extraction [J]. Computer and Modernization, 2022, 0(06): 21-26.
[13]	FAN Hai-wei, QIN Jia-jie, SUN Huan, ZHANG Li-miao, LU Xin-siyu. Traffic Accident Text Information Extraction Model Based on BERT and BiGRU-CRF Fusion [J]. Computer and Modernization, 2022, 0(05): 10-15.
[14]	GUO Tian-yu, YAN Rong-guo, FANG Xu-chen, XU Yu-ling, TAO Zheng-yi. Detection of R Wave Based on Hilbert Transform and Adaptive Threshold [J]. Computer and Modernization, 2022, 0(02): 114-119.
[15]	LIU Meng-ying, WANG Yong. Microblog Hot Topic Discovery Based on Text Dual Representation Model [J]. Computer and Modernization, 2021, 0(12): 110-115.