An Automatic Text Summarization Model Construction Method Based on BERT Embedding

doi:10.3969/j.issn.1006-2475.2020.01.012

Abstract

Abstract: Aiming at the problem that the traditional word vector can not effectively represent polysemous words in text summarization, which reduces the accuracy and readability of summarization, this paper proposes an automatic text summarization model construction method based on BERT (Bidirectional Encoder Representations from Transformers)Embedding. This method introduces the BERT pre-training language model to enhance the semantic representation of word vector. The generated word vectors are input into the Seq2Seq model for training to form an automatic text summarization model, which realizes the rapid generation of text summarization. The experimental results show that the model can effectively improve the accuracy and readability of the generated summarization on Gigaword dataset, and can be used for automatic text summarization generation tasks.

Key words: text summarization, BERT model, attention mechanism, Sequence-to-Sequence(Seq2Seq) model

CLC Number:

TP391.1

YUE Yi-feng, HUANG Wei, REN Xiang-hui. An Automatic Text Summarization Model Construction Method Based on BERT Embedding[J]. Computer and Modernization, doi: 10.3969/j.issn.1006-2475.2020.01.012.

References

［1］李慕. 结合自动摘要技术的文本推荐方法研究及应用［D］. 武汉：武汉工程大学, 2017.
［2］沈华东，彭敦陆. AM-BRNN：一种基于深度学习的文本摘要自动抽取模型［J］. 小型微型计算机系统, 2018，39（6）：1184-1189.
［3］ BHASKAR P, BANDYOPADHYAY S. A query focused multi document automatic summarization［C］// Proceedings of the 24th Pacific Asia Conference on Language, Information and Computation. 2010:545-554.
［4］叶静. 面向多文本集的比较摘要研究［D］. 长沙：国防科学技术大学, 2012.
［5］ MIHALCEA R. Graph-based ranking algorithms for sentence extraction, applied to text summarization［C］// Proceedings of the ACL Interactive Poster and Demonstration Sessions. 2004:170-173.
［6］ BADRINATH R, VENKATASUBRAMANIYAN S, MADHAVAN C E V. Improving query focused summarization using look-ahead strategy［C］// European Conference on Information Retrieval. 2011:641-652.
［7］ YANG L, CAI X. Semi-supervised co-clustering for query-oriented theme-based summarization［J］. Research Journal of Applied Sciences, Engineering and Technology, 2012,4(18):3410-3414.
［8］ RUSH A M, CHOPRA S, WESTON J. A neural attention model for abstractive sentence summarization［J/OL］. 2015, arXiv: 1509.00685, (2015-09-03)［2019-06-01］. https://arxiv.org/pdf/1509.00685.pdf.
［9］ DEVLIN J, CHANG M W, LEE K, et al. Bert: Pre-training of deep bidirectional transformers for language understanding［J/OL］. 2018, arXiv: 1810.04805, (2018-10-11)［2019-06-01］. https:/arxiv.org/abs/1810.04805.
［10］VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need［C］// Advances in Neural Information Processing Systems. 2017:5998-6008.
［11］SUTSKEVER I, VINYALS O, LE Q V. Sequence to sequence learning with neural networks［C］// Advances in Neural Information Processing Systems. 2014:3104-3112.
［12］SENNRICH R, HADDOW B. Linguistic input features improve neural machine translation［J/OL］. 2016, arXiv: 1606.02892, (2016-07-27)［2019-06-01］. https://arxiv.org/pdf/1606.0289v2.pdf.
［13］蒲梅,周枫,周晶晶,等. 基于加权TextRank的新闻关键事件主题句提取［J］. 计算机工程, 2017,34(8):219-224.
［14］MIKOLOV T, SUTSKEVER I, CHEN K, et al. Distributed representations of words and phrases and their compositionality［C］// Advances in Neural Information Processing Systems. 2013:3111-3119.
［15］洪冬梅. 基于LSTM的自动文本摘要技术研究［D］. 广州：华南理工大学, 2018.
［16］唐晓波,翟夏普. 基于混合机器学习模型的多文档自动摘要［J］. 情报理论与实践, 2019，42(2):145-150.
［17］NEMA P, KHAPRA M, LAHA A, et al. Diversity driven attention model for query-based abstractive summarization［J/OL］. 2017, arXiv:1704.08300, (2018-06-07)［2019-06-01］. https://arxiv.org/pdf/1704.08300.pdf.
［18］明拓思宇，陈鸿昶. 文本摘要研究进展与趋势［D］. 郑州：国家数字交换系统工程技术研究中心, 2018.
［19］官宸宇. 面向事件的社交媒体文本自动摘要研究［D］. 武汉：武汉大学, 2017.
［20］郭捷. 基于网络评论的情感分类技术的研究及应用［D］. 成都：电子科技大学, 2018.
［21］徐立鑫. 面向短文本流摘要抽取系统的在线学习技术［D］. 北京：北京邮电大学, 2015.
［22］MEMISEVIC R, ZACH C, POLLEFEYS M, et al. Gated softmax classification［C］// Advances in Neural Information Processing Systems. 2010:1603-1611.
［23］LIN C Y, HOVY E. Automatic evaluation of summaries using n-gram co-occurrence statistics［C］// Proceedings of 2003 Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics. 2003:150-157.
［24］〖JP2〗PENNINGTON J, SOCHER R, MANNING C. Glove: Global vectors for word representation［C］// Proceedings of 2014 Conference on Empirical Methods in Natural Language Processing. 2014:1532-1543.

[1]	WANG Qiu-yi, ZHOU Hao, ZHENG Ting-ting. Improved RetinaNet Target Detection Method for Power Equipment [J]. Computer and Modernization, 2024, 0(01): 47-52.
[2]	QIU Kai-xing, FENG Guang. A Multi-label Image Classification Model Based on Dual Feature Attention [J]. Computer and Modernization, 2023, 0(12): 41-47.
[3]	ZHANG Hao-yang, YIN Zi-ming, LE Jun-yi, SHEN Da-cong, SHU Yi-jun, YANG Zi-yi, . 3D-SPRNet: Segmentation Model of Gallbladder Cancer Based on Parallel Decoder and Double Attention Mechanism [J]. Computer and Modernization, 2023, 0(12): 59-66.
[4]	ZHANG Bo-quan, MAI Hai-peng, CHEN Jia-min, Pang Jin-ju. White Matter Hyperintensities Segmentation Based on High Gray Value#br# Attention Mechanism [J]. Computer and Modernization, 2023, 0(12): 67-75.
[5]	WANG Yu-hang, DONG Bao-liang, GONG Chao, SHANG Zhen-zhen, YAO Kang-ning. Dynamic Threat Assessment of Air Swarm Targets Based on Intent Recognition [J]. Computer and Modernization, 2023, 0(12): 100-104.
[6]	LUO Ming-jie, FENG Kai-ping. Lightweight Facial Expression Recognition Method Based on Sandglass Structure and Attention Mechanism [J]. Computer and Modernization, 2023, 0(11): 89-94.
[7]	ZHANG Jia-Qi, XU Qi-lei. Apple Defect Detection Algorithm Based on NAM-YOLO Network [J]. Computer and Modernization, 2023, 0(10): 53-58.
[8]	YE Si-jia, WEI Yan, DU Han-yu, DENG Jin-zhi. HRNet Image Semantic Segmentation Algorithm Combined with Attention Mechanism [J]. Computer and Modernization, 2023, 0(10): 65-69.
[9]	CHEN Jia-min, ZHANG Bo-quan, MAI Hai-peng. Hippocampus Segmentation Based on Feature Fusion [J]. Computer and Modernization, 2023, 0(08): 1-6.
[10]	LIU Xu, ZHA Ke-ke. An Environmental Target Recognition Method for Airport Special Vehicle Operation [J]. Computer and Modernization, 2023, 0(08): 18-24.
[11]	WANG Hong, GE Hong. Cross Modal Hash Retrieval Based on Attention Mechanism and Semantic Similarity [J]. Computer and Modernization, 2023, 0(08): 44-53.
[12]	OUYANG Fei, WU Xu, XIANG Dong-sheng. Garbage Classification and Detection Method Based on Improved YOLOX [J]. Computer and Modernization, 2023, 0(08): 68-73.
[13]	CUI Shao-guo, ZHANG Gang, WANG Ao-di. Deep Cross Network Recommendation Model Based on Attention Perception [J]. Computer and Modernization, 2023, 0(07): 54-60.
[14]	QIN Zhu-yuan, WU Hao-zhong, TAN Dai-qing, HAN Ai-qing, ZANG Hao, WANG Xuan, TANG Yan. Fine-grained Identification of Maidong Based on Multi-scale ResNet Combining Attention Mechanism [J]. Computer and Modernization, 2023, 0(07): 105-111.
[15]	GONG Xuan, GUO Zhong-hua, CHEN Wang. Remote Sensing Image Road Segmentation Based on CA-TransUNet [J]. Computer and Modernization, 2023, 0(07): 112-118.