Machine Translation System Based on Self-Attention Model

doi:10.3969/j.issn.1006-2475.2019.07.002

Abstract

Abstract: In recent years, neural machine translation (NMT) has developed rapidly. The proposed Seq2Seq framework brings great advantages to machine translation. It can generate arbitrary output sequences after observing the entire input sentence. However, this model still has great limitations on the ability to capture long-distance information. The proposed recurrent neural network (RNN) and LSTM network were all proposed to improve this problem, but the effect is not obvious. The presentation of the attention mechanism effectively compensates for this deficiency. The Self-Attention model is proposed on the basis of attention mechanism, and an encoder-decoder framework is built based on Self-Attention. This paper explores the previous neural network translation model. The mechanism and principle of the Self-Attention model are analyzed. The translation system is realized based on Self-Attention model by TensorFlow deep learning framework. In the English-to-Chinese translation experiment, compared with the previous neural network translation model, it shows that the model has a good translation effect.

Key words: neural machine translation, Seq2Seq, attention mechanism, Self-Attention model

CLC Number:

TP391

SHI Yan, WANG Yu, WU Shui-qing. Machine Translation System Based on Self-Attention Model[J]. Computer and Modernization, doi: 10.3969/j.issn.1006-2475.2019.07.002.

References

［1］张家俊,宗成庆. 神经网络语言模型在统计机器翻译中的应用［J］. 情报工程, 2017,3(3):21-28.
［2］刘洋. 神经机器翻译前沿进展［J］. 计算机研究与发展, 2017,54(6):1144-1149.
［3］ SUTSKEVER I, VINYALS O, LE Q V. Sequence to sequence learning with neural networks［C］// Advances in Neural Information Processing Systems 27 (NIPS 2014). 2014:3104-3112.
［4］ BAHDANAU D, CHO K H, BENGIO Y. Neural Machine Translation by Jointly Learning to Align and Translate［J/OL］. (2014-12-19)［2018-12-10］. https://arxiv.org/pdf/1409.0473v4.pdf.
［5］ KALCHBRENNER N, BLUNSOM P. Recurrent continuous translation models［C］// Proceedings of the 2013 ACL Conference on Empirical Methods in Natural Language Processing (EMNLP). 2013:1700-1709.
［6］ CHO K H, VAN MERRIENBOER B, BAHDANAU D, et al. On the properties of neural machine translation: Encoder-Decoder approaches［C］// Proceedings of the 8th Workshop on Syntax, Semantics and Structure in Statistical Translation. 2014:103-111.
［7］ DYER C, KUNCORO A, BALLESTEROS M, et al. Recurrent neural network grammars［C］// Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2016:199-209.
［8］ CHUNG J Y, GULCEHRE , CHO K H, et al. Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling［J/OL］. (2014-12-11)［2018-12-10］. https://arxiv.org/pdf/1412.3555.pdf.
［9］ GULCEHRE , FIRAT O, XU K, etal. On Using Monolingual Corpora in Neural Machine Translation［J/OL］. (2015-06-12)［2018-12-10］. https://arxiv.org/pdf/1503.03535.pdf.
［10］WU Y H, SCHUSTER M, CHEN Z F, et al. Google’s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation［J/OL］. (2016-09-26)［2018-12-10］. https://arxiv.org/pdf/1609.08144v1.pdf.
［11］PASCANU A, MIKOLOV T, BENGIO Y. On the Difficulty of Training Recurrent Neural Networks［J/OL］. (2013-02-16)［2018-12-10］. https://arxiv.org/pdf/1211.5063.pdf.
［12］HOCHREITER S, BENGIO Y, FRASCONI P, et al. Gradient flow in recurrent nets: The difficulty of learning long-term dependencies［M］// A Field Guide to Dynamical Recurrent Neural Networks. Wiley, 2001:237-243.
［13］〖JP+2〗HOCHREITER S, SCHMIDHUBER J. Long short-term memory［J］. Neural Computation, 1997,9(8):1735-1780.
［14］HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition［C］// Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. 2016:770-778.
［15］KIM Y, DENTON C, HOANG L, et al. Structured Attention Networks［J/OL］. (2017-02-16)［2018-12-10］. https://arxiv.org/pdf/1702.00887.pdf.
［16］LUONG M T, PHAM H, MANNING C D. Effective Approaches to Attention-based Neural Machine Translation［J/OL］. (2015-09-20)［2018-12-10］. https://arxiv.org/pdf/1508.04025.pdf.
［17］VASWANI A, SHAZEER N, PARMAR N, et al. Attention Is All You Need［J/OL］. (2017-06-30)［2018-12-10］. https://arxiv.org/pdf/1706.03762v4.pdf.
［18］BRITZ D, GOLDIE A, LUONG M T, et al. Massive Exploration of Neural Machine Translation Architectures［J/OL］. (2017-03-21)［2018-12-10］. https://arxiv.org/pdf/1703.03906.pdf.
［19］CHO K H, VAN MERRIENBOER B, GULCEHRE , et al. Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation［J/OL］. (2014-09-03)［2018-12-10］. https://arxiv.org/pdf/1406.1078.pdf.
［20］KAISER L, BENGIO S. Can Active Memory Replace Attention?［J/OL］. (2016-10-27)［2018-12-10］. https://arxiv.org/pdf/1610.08613v1.pdf.
［21］BA J L, KIROS J R, HINTON G E. Layer Normalization［J/OL］. (2016-07-21)［2018-12-10］. https://arxiv.org/pdf/1607.06450.pdf.
［22］GEHRING J, AULI M, GRANGIER D, et al. Convolutional Sequence to Sequence Learning［J/OL］. (2017-05-12)［2018-12-10］. https://arxiv.org/pdf/1705.03122v2.pdf.
［23］PAPINENI K, ROUKOS S, WARD T, et al. BLEU: A method for automatic evaluation of machine translation［C］// Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics. 2002,7:311-318.

[1]	WANG Qiu-yi, ZHOU Hao, ZHENG Ting-ting. Improved RetinaNet Target Detection Method for Power Equipment [J]. Computer and Modernization, 2024, 0(01): 47-52.
[2]	QIU Kai-xing, FENG Guang. A Multi-label Image Classification Model Based on Dual Feature Attention [J]. Computer and Modernization, 2023, 0(12): 41-47.
[3]	ZHANG Hao-yang, YIN Zi-ming, LE Jun-yi, SHEN Da-cong, SHU Yi-jun, YANG Zi-yi, . 3D-SPRNet: Segmentation Model of Gallbladder Cancer Based on Parallel Decoder and Double Attention Mechanism [J]. Computer and Modernization, 2023, 0(12): 59-66.
[4]	ZHANG Bo-quan, MAI Hai-peng, CHEN Jia-min, Pang Jin-ju. White Matter Hyperintensities Segmentation Based on High Gray Value#br# Attention Mechanism [J]. Computer and Modernization, 2023, 0(12): 67-75.
[5]	WANG Yu-hang, DONG Bao-liang, GONG Chao, SHANG Zhen-zhen, YAO Kang-ning. Dynamic Threat Assessment of Air Swarm Targets Based on Intent Recognition [J]. Computer and Modernization, 2023, 0(12): 100-104.
[6]	LUO Ming-jie, FENG Kai-ping. Lightweight Facial Expression Recognition Method Based on Sandglass Structure and Attention Mechanism [J]. Computer and Modernization, 2023, 0(11): 89-94.
[7]	ZHANG Jia-Qi, XU Qi-lei. Apple Defect Detection Algorithm Based on NAM-YOLO Network [J]. Computer and Modernization, 2023, 0(10): 53-58.
[8]	YE Si-jia, WEI Yan, DU Han-yu, DENG Jin-zhi. HRNet Image Semantic Segmentation Algorithm Combined with Attention Mechanism [J]. Computer and Modernization, 2023, 0(10): 65-69.
[9]	CHEN Jia-min, ZHANG Bo-quan, MAI Hai-peng. Hippocampus Segmentation Based on Feature Fusion [J]. Computer and Modernization, 2023, 0(08): 1-6.
[10]	LIU Xu, ZHA Ke-ke. An Environmental Target Recognition Method for Airport Special Vehicle Operation [J]. Computer and Modernization, 2023, 0(08): 18-24.
[11]	WANG Hong, GE Hong. Cross Modal Hash Retrieval Based on Attention Mechanism and Semantic Similarity [J]. Computer and Modernization, 2023, 0(08): 44-53.
[12]	OUYANG Fei, WU Xu, XIANG Dong-sheng. Garbage Classification and Detection Method Based on Improved YOLOX [J]. Computer and Modernization, 2023, 0(08): 68-73.
[13]	CUI Shao-guo, ZHANG Gang, WANG Ao-di. Deep Cross Network Recommendation Model Based on Attention Perception [J]. Computer and Modernization, 2023, 0(07): 54-60.
[14]	QIN Zhu-yuan, WU Hao-zhong, TAN Dai-qing, HAN Ai-qing, ZANG Hao, WANG Xuan, TANG Yan. Fine-grained Identification of Maidong Based on Multi-scale ResNet Combining Attention Mechanism [J]. Computer and Modernization, 2023, 0(07): 105-111.
[15]	GONG Xuan, GUO Zhong-hua, CHEN Wang. Remote Sensing Image Road Segmentation Based on CA-TransUNet [J]. Computer and Modernization, 2023, 0(07): 112-118.