基于改进对抗学习及融合特征的短文本分类框架

doi:10.3969/j.issn.1006-2475.2024.04.012

摘要/Abstract

摘要： 摘要：在短文本分类中，由于短文本具有字数较少、存在歧义、关键信息少并且不易捕捉等特点，对其进行分类的算法模型在训练及推理时往往存在差异性，并且主流分类模型基本是在关键特征上进行建模而忽略了非关键特征信息，这加大了其在精确分类上面临的挑战。针对上述问题，本文提出一种结合多种对抗训练策略融合及改进自注意力机制的短文本分类框架。模型一开始在文本向量表示层面进行对字嵌入加入对抗扰动，强化文本表示能力，并在F1分数达到一定阈值后加入改进对模型权重的对抗扰动，强化模型在训练及推理时的泛化能力，从而辅助提高该框架各分类器的特征学习能力。在特征学习网络模块方面，本文利用多尺度卷积模块和双向长短期记忆神经网络相结合来学习不同粒度特征，为了学习不相邻的特征信息，引入空洞卷积，增大卷积感受野，设计一个门控机制来控制该层信息的学习速度；最后，通过加入一种新型注意力机制，建模关键信息的同时也建模了非关键特征信息，同时加入损失进行计算，增强模型学习特征信息的能力并降低过拟合的风险。在2个大型公开数据集THUCNews新闻标题数据集及今日头条新闻标题数据集测试显示，本文方法的F1分数相比于当前主流模型及经典模型分类效果最多提升了4.93个百分点及6.14个百分点，取得了不错的效果，本文还对加入权重扰动阈值和不同模块的有效性进行了对比及消融实验探究。

关键词: 关键词：对抗训练, 策略融合, 空洞卷积, 非关键信息, 注意力机制

Abstract: Abstract: Text classification is one of the most important directions in natural language processing research. Short text has the characteristics of less word count， ambiguity， less key information and not easy to capture， but the algorithm models that classify them are often different in training and reasoning， and mainstream classification models basically model key features and ignore non-key feature information， which increases the challenges in accurate classification. In order to solve the above problems， this paper proposes a short text classification framework combining the fusion of multiple adversarial training strategies and improving the self-attention mechanism. At the beginning， the model adds adversarial perturbation to the text vector representation level to strengthen the text representation ability， and adds an adversarial perturbation to improve the model weights after the F1 score reaches a certain threshold to strengthen the generalization ability of the model during training and inference， thereby assisting in improving the feature learning ability of each classifier of the framework. In terms of feature learning network module， this paper uses the combination of multi-scale convolutional module and bidirectional long short-term memory neural network to learn different granular features， in order to learn nonadjacent feature information， introduces hole convolution， increases the convolution receptive field， and designs a gating mechanism to control the learning speed of this layer information. Finally， by adding a new attention mechanism， the key information is modeled and the non-critical information is modeled， and the loss is added for calculation， which enhances the model’s ability to learn feature information and reduces the risk of overfitting. The tests of THUCNews news headline dataset and Toutiao headline dataset of two large-scale public datasets show that the F1 score of this method is increased by up to 4.93 percentage points and 6.14 percentage points compared with the current mainstream model and classical model， and the effectiveness of adding weight disturbance threshold and different modules is also compared and ablation experiments are explored.

Key words: Key words: adversarial training, strategy fusion, void convolution, non-critical information, attention mechanisms

中图分类号:

TP391

宁召阳1, 2, 申情2, 3, 郝秀兰1, 2, 赵康1, 2. 基于改进对抗学习及融合特征的短文本分类框架[J]. 计算机与现代化, 2024, 0(04): 66-76.

NING Zhaoyang1, 2, SHEN Qing2, 3, HAO Xiulan1, 2, ZHAO Kang1, 2. Short Text Classification Method Based on Improved Adversarial Learning and Fusion Features[J]. Computer and Modernization, 2024, 0(04): 66-76.

参考文献

［1］ MAAS A L， DALY R E， PHAM P T， et al. Learning word vectors for sentiment analysis［C］// Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies. ACL， 2011:142-150.
［2］ MIKOLOV T， SUTSKEVER I， CHEN K， et al. Distributed representations of words and phrases and their compositionality［C］// Proceedings of the 26th International Conference on Neural Information Processing Systems. Curran and Associates Inc， 2013:3111-3119.
［3］ MIKOLOV T， CHEN K， CORRADO G， et al. Efficient estimation of word representations in vector space［J］. arXiv preprint arXiv:1301.3781， 2013.
［4］ JOULIN A， GRAVE E， BOJANOWSKI P， et al. Bag of tricks for efficient text classification［C］// Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics. ACL， 2017:427-431.
［5］ PENNINGTON J， SOCHER R， MANNING C D. GloVe: Global vectors for word representation［C］// Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing （EMNLP）. ACL， 2014:1532-1543.
［6］ DEVLIN J， CHANG M W， LEE K T， et al. BERT: Pre-training of deep bidirectional transformers for language understanding［C］// Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. ACL， 2019:4171-4186.
［7］ LAN Z Z， CHEN M D， GOODMAN S， et al. ALBERT: A lite BERT for self-supervised learning of language representations［J］. arXiv preprint arXiv:1909.11942， 2019.
［8］ HE P C， LIU X D， GAO J F， et al. DeBERTa: Decoding-enhanced BERT with disentangled attention［J］. arXiv preprint arXiv:2006.03654， 2020.
［9］ KIM Y. Convolutional neural networks for sentence classification［C］// Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing （EMNLP）. ACL， 2014:1746-1751.
［10］ MIKOLOV T， KOMBRINK S， BURGET L， et al. Extensions of recurrent neural network language model［C］// Proceedings of the 2011 IEEE International Conference on Acoustics， Speech and Signal Processing. IEEE， 2011:5528-5531.
［11］ SOCHER R， LIN C C Y， NG A Y， et al. Parsing natural scenes and natural language with recursive neural networks［C］// Proceedings of the 28th International Conference on Machine Learning. Omnipress， 2011:129-136.
［12］何力，郑灶贤，项凤涛，等. 基于深度学习的文本分类技术研究进展［J］. 计算机工程， 2021，47（2）:1-11.
［13］ CHO K， VAN MERRIENBOER B， GULCEHRE C， et al. Learning phrase representations using RNN encoder-decoder for statistical machine translation［C］// Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing （EMNLP）. ACL， 2014:1724-1734.
［14］ LAI S W， XU L H， LIU K， et al. Recurrent convolutional neural networks for text classification［C］// Proceedings of the 29th AAAI Conference on Artificial Intelligence. AAAI Press， 2015，29（1）:2267-2273.
［15］ DEY R， SALEM F M. Gate-variants of gated recurrent unit （GRU） neural networks［C］// Proceedings of the 60th IEEE International Midwest Symposium on Circuits and Systems. IEEE， 2017:1597-1600.
［16］ YAO L， MAO C S， LUO Y. Graph convolutional networks for text classification［C］// Proceedings of the 33rd AAAI Conference on Artificial Intelligence. AAAI Press， 2019，33（1）:7370-7377.
［17］ KIPF T N， WELLING M. Semi-supervised classification with graph convolutional networks［J］. arXiv preprint arXiv:1609.02907， 2016.
［18］ QIN Q， HU W P， LIU B. Feature projection for improved text classification［C］// Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. ACL， 2020:8161-8171.
［19］程艳，尧磊波，张光河，等. 基于注意力机制的多通道CNN和BiGRU的文本情感倾向性分析［J］. 计算机研究与发展， 2020，57（12）:2583-2595.
［20］ ZHOU P， SHI W， TIAN J， et al. Attention-based bidirectional long short-term memory networks for relation classification［C］// Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics. ACL， 2016:207-212.
［21］李博涵，向宇轩，封顶，等. 融合知识感知与双重注意力的短文本分类模型［J］. 软件学报， 2022，33（10）:3565-3581.
［22］ GOODFELLOW I J， SHLENS J， SZEGEDY C. Explaining and harnessing adversarial examples［J］. arXiv preprint arXiv:1412.6572， 2014.
［23］ MIYATO T， DAI A M， GOODFELLOW I. Adversarial training methods for semi-supervised text classification［J］. arXiv preprint arXiv:1605.07725， 2016.
［24］ HE K M， FAN H Q， WU Y X， et al. Momentum contrast for unsupervised visual representation learning［C］// Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition （CVPR）. IEEE， 2020:9726-9735.
［25］ YE M， ZHANG X， YUEN P C， et al. Unsupervised embedding learning via invariant and spreading instance feature［C］// Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition （CVPR）. IEEE， 2019:6203-6213.
［26］ GAO T Y， YAO X C， CHEN D Q. SimCSE: Simple contrastive learning of sentence embeddings［C］// Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing （EMNLP）. ACL， 2021:6894-6910.
［27］ LIANG X B， WU L J， LI J T， et al. R-Drop: Regularized dropout for neural networks［J］. arXiv preprint arXiv:2106.14448， 2021.
［28］ WU D X， XIA S T， WANG Y S. Adversarial weight perturbation helps robust generalization［J］. arXiv preprint arXiv:2004.05884， 2020.
［29］ YU F， KOLTUN V. Multi-Scale context aggregation by dilated convolutions［J］. arXiv preprint arXiv:1511.07122， 2015.
［30］ VASWANI A， SHAZEER N， PARMAR N， et al. Attention is all you need［C］// Advances in Neural Information Processing Systems 30 （NIPS 2017）. Curran Associates Inc， 2017:6000-6010.
［31］ KINGMA D P， BA J. Adam: A method for stochastic optimization［J］. arXiv preprint arXiv:1412.6980， 2014.
［32］ JOHNSON R， ZHANG T. Deep pyramid convolutional neural networks for text categorization［C］// Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics. ACL， 2017:562-570.
［33］张海丰，曾诚，潘列，等. 结合BERT和特征投影网络的新闻主题文本分类方法［J］. 计算机应用， 2022，42（4）:1116-1124.
［34］刘硕，王庚润，彭建华，等. 基于混合字词特征的中文短文本分类算法［J］. 计算机科学， 2022，49（4）:282-287.
［35］李启行，廖薇，孟静雯. 基于注意力机制的双通道DAC-RNN文本分类模型［J］. 计算机工程与应用， 2021，58（16）:157-163.
［36］范昊，何灏. 融合上下文特征和BERT词嵌入的新闻标题分类研究［J］. 情报科学， 2022，40（6）:90-97.

[1]	高埂1, 肖风丽2, 杨飞1. 基于改进MobileNetV3-Small的色素减退性皮肤病诊断[J]. 计算机与现代化, 2024, 0(05): 120-126.
[2]	崔少国, 张宇楠. 融合多尺度空间特征的甲状腺结节超声图像分割[J]. 计算机与现代化, 2024, 0(03): 72-77.
[3]	刘学虎, 欧鸥, 张伟劲, 杜雪垒. 融合注意力机制和空洞卷积的滑坡图像检测[J]. 计算机与现代化, 2022, 0(04): 45-51.
[4]	胡昌冉, 樊彦国, 禹定峰. 嵌入空洞卷积模块的改进YOLOv3车辆检测算法[J]. 计算机与现代化, 2021, 0(04): 53-60.
[5]	周幸, 陈立福. 基于双注意力机制的遥感图像目标检测[J]. 计算机与现代化, 2020, 0(08): 1-7.