Text Matching Model Based on BERT and Self-attention Mechanism of Image

Abstract

Abstract: In order to improve the accuracy of text matching, a text matching model based on BERT(Bidirectional Encoder Representations from Transformers) and self-attention mechanism of image is proposed to overcome the limitations of BERT model and MatchPyramid model in text matching. Firstly, a pair of text is encoded into word-level feature vectors by using the BERT model. Secondly, the matching matrix of word to word similarity between two texts is constructed according to the word vector, which is regarded as a single channel image representation. Then the self-attention feature matrix of the matching matrix is generated by the self-attention mechanism of image. Finally, the matching matrix and the self-attention feature matrix are connected into multi-channel to capture the text matching signals in the image by the convolutional neural network. After the matching signal is connected with the coding vector called ［CLS］ which is yielded by the BERT model, the similarity of the two texts is obtained by inputting the fully connected neural layer. The experimental results show that the model can effectively improve the MAP and MRR metrics compared with BERT model, MatchPyramid model and other text matching models on WikiQA dataset, and the effectiveness of the model is verified.

Key words: matching matrix, self-attention mechanism of image, feature fusion, text matching, BERT model

SONG Shuang, LU Xin-da. Text Matching Model Based on BERT and Self-attention Mechanism of Image[J]. Computer and Modernization, 2021, 0(11): 12-16.

References

［1］中国互联网络信息中心. CNNIC发布第47次《中国互联网络发展状况统计报告》［EB/OL］. （2021-02-03)［2021-02-03］. http://cnnic.cn/gywm/xwzx/rdxw/20172017_ 7084/202102/t20210203_71364.htm.
［2］陈肇雄，高庆狮. 自然语言处理［J］. 计算机研究与发展, 1989(11):1-16.
［3］ LIN D K, PANTEL P. Discovery of inference rules for question-answering［J］. Natural Language Engineering, 2001,7(4):343-360.
［4］ LI H, XU J. Semantic matching in search［J］. Foundations and Trends in information Retrieval, 2014,7(5):343-469.
［5］高璐璐,赵雯. 机器翻译研究综述［J］. 中国外语, 2020,17(6):97-103.
［6］ YIN W P, SCHUTZE H. Convolutional neural network for paraphrase identification［C］// Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2015:901-911.
［7］ FENG M W, XIANG B, GLASS M R, et al. Applying deep learning to answer selection: A study and an open task［C］// Proceedings of 2015 IEEE Workshop on Automatic Speech Recognition and Understanding. 2015:813-820.
［8］朱大奇. 人工神经网络研究现状及其展望［J］. 江南大学学报(自然科学版), 2004,3(1):103-110.
［9］陈珂,梁斌,柯文德,等. 基于多通道卷积神经网络的中文微博情感分析［J］. 计算机研究与发展, 2018,55(5):945-957.
［10］HU B T, LU Z D, LI H, et al. Convolutional neural network architectures for matching natural language sentences［C］// Proceedings of the 2014 Advances in Neural Information Processing Systems. 2014:2042-2050.
［11］PANG L, LAN Y Y, GUO J F, et al. Text matching as image recognition［C］// Proceedings of the 30th AAAI Conference on Artificial Intelligence. 2016:2793-2799.
［12］LECUN Y, BENGIO Y. Convolutional networks for images, speech, and time series［M］// The Handbook of Brain Theory and Neural Networks.MIP Press, Cambridge, 1998:255-258.
［13］SZEGEDY C, LIU W, JIA Y Q, et al. Going deeper with convolutions［C］// Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition. 2015:1-9.

［14］张琦,张荣梅,陈彬. 基于深度学习的图像识别技术研究综述［J］. 河北省科学院学报, 2019,36(3):28-36.

［15］丁旭甫,王宏生. 基于深度学习的图像识别技术的研究［J］. 信息与电脑(理论版), 2019(7):124-125.
［16］DEVLIN J, CHANG M W, LEE K, et al. BERT: Pre-training of deep bidirectional transformers for language understanding［J］. arXiv preprint arXiv:1810.04805, 2018.
［17］LAN Z Z, CHEN M D, GOODMAN S, et al. ALBERT: A lite BERT for self-supervised learning of language representations［J］. arXiv preprint arXiv:1909.11942, 2019.
［18］ZHANG Z S, WU Y W, ZHAO H, et al. Semantics-aware BERT for language understanding［C］// Proceedings of the 34th AAAI Conference on Artificial Intelligence. 2020:9628-9635.
［19］ABBAS F, MALIK M K, RASHID M U, et al. WikiQA: A question answering system on Wikipedia using freebase, DBpedia and Infobox［C］// Proceedings of the 6th IEEE International Conference on Innovative Computing Technology. 2016:185-193.
［20］YANG Y, YIH W T, MEEK C. WikiQA: A challenge dataset for open-domain question answering［C］// Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. 2015:2013-2018.
［21］ROCKTASCHEL T, GREFENSTETTE E, HERMANN K, et al. Reasoning about entailment with neural attention［C］// Proceedings of the 4th International Conference on Learning Representations. 2016:716-723.
［22］YIN W P, SCHUTZE H. MultiGranCNN: An architecture for general matching of text chunks on multiple levels of granularity［C］// Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics & the 7th International Joint Conference on Natural Language Processing. 2015:63-73.
［23］YIN W P, SCHUTZE H, XIANG B, et al. ABCNN: Attention-based convolutional neural network for modeling sentence pairs［J］. Transactions of the Association for Computational Linguistics, 2016,4:259-272.
［24］TYMOSHENKO K, MOSCHITTI A. Cross-pair text representations for answer sentence selection［C］// Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. 2018:2162-2173.
［25］WANG Z G, HAMZA W, FLORIAN R. Bilateral multi-perspective matching for natural language sentences［C］// Proceedings of the 26th International Joint Conference on Artificial Intelligence. 2017:4144-4150.
［26］TAY Y, TUAN A L, HUI S C. Hyperbolic representation learning for fast and efficient neural question answering［C］// Proceedings of the 11th ACM International Conference on Web Search and Data Mining. 2018:583-591.
［27］SHAO B, GONG Y Y, QI W Z, et al. Aggregating bidirectional encoder representations using matchLSTM for sequence matching［C］// Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing. 2019:6059-6063.

[1]	ZHANG Simin, LIU Xinmei, YIN Junling, LI Baoling. PCB Defect Detection Method Based on Improved YOLOv7 [J]. Computer and Modernization, 2024, 0(12): 45-52.
[2]	WANG Haiyang, GONG Tongxin, YANG Jintao, CHEN Zailong. Short-term Load Forecasting in Industrial Parks with Multi-scale Time Coding [J]. Computer and Modernization, 2024, 0(12): 59-65.
[3]	MA Yu, YANG Yong, REN Ge, Palidan Tuerxun. Automated Essay Scoring Method Based on GCN and Fine Tuned BERT [J]. Computer and Modernization, 2024, 0(09): 33-37.
[4]	ZHENG Shangpo1, CHEN Defu1, LI Jianli2, LIN Guoxian2, WANG Xingping3. Pedestrian Tracking Algorithm Based on Improved YOLOv5s and DeepSORT [J]. Computer and Modernization, 2024, 0(08): 54-58.
[5]	PANG Mei, WANG Gong, ZHAN Yong, HUANG Zhefa. Underwater Trash Detection Method Based on Improved YOLOv5 [J]. Computer and Modernization, 2024, 0(07): 120-126.
[6]	FU Lingli, QIU Yu, ZHANG Xinchen. Retinal Vessel Segmentation Based on Improved U-Net with Multi-feature Fusion [J]. Computer and Modernization, 2024, 0(06): 76-82.
[7]	ZHU Fen, HE Lifeng, SUN Shuang, ZHANG Mengying, YU Jiajia. Pancreas Segmentation Model Based on Deformable Residual and Cascading Encoding [J]. Computer and Modernization, 2024, 0(06): 83-88.
[8]	WU Zhao-meng1, ZHANG Cheng-gang2. Unpaired Cross-modal Hashing Method for Web News Data [J]. Computer and Modernization, 2024, 0(03): 54-60.
[9]	NING Juan, ZHOU Qing-hua, ZENG Xiao-wei. Application of Improved YOLOv7 Algorithm in Detection of Capping Defects of Vials [J]. Computer and Modernization, 2023, 0(12): 82-86.
[10]	CHEN Jun-yi. Health Event Prediction Model Based on Dynamic and Static Features of Graph Nodes [J]. Computer and Modernization, 2023, 0(10): 39-44.
[11]	XING Shi-shuai, LIU Dan-feng, WANG Li-guo, PAN Yue-tao, MENG Ling-hong, YUE Xiao-han. Image Super-resolution Reconstruction Based on Spatial Attention Residual Network [J]. Computer and Modernization, 2023, 0(10): 45-52.
[12]	CHEN Jia-min, ZHANG Bo-quan, MAI Hai-peng. Hippocampus Segmentation Based on Feature Fusion [J]. Computer and Modernization, 2023, 0(08): 1-6.
[13]	WANG Hong, GE Hong. Cross Modal Hash Retrieval Based on Attention Mechanism and Semantic Similarity [J]. Computer and Modernization, 2023, 0(08): 44-53.
[14]	WANG Jie, PAN Feng, ZHANG Yan-sha, TAN Mian, YAN Xiao-bo, WANG Lin, . Surface Defect Classification of Aluminum Profiles with Weighted Non-local Modules [J]. Computer and Modernization, 2023, 0(05): 86-92.
[15]	ZHU Li-qing, LI Xiang, . Vehicle Detection of Remote Sensing Images Based on Improved YOLOv5 Algorithm [J]. Computer and Modernization, 2023, 0(05): 117-121.