关键词: 注意力机制, 卷积神经网络, 长短时记忆网络(LSTM), 图像描述

Abstract: Attention-based neural encoder-decoder frameworks have been widely adopted for image captioning. Most methods force visual attention to be active for every generated word. However, the decoder likely requires little to no visual information from the image to predict nonvisual words such as “the” and “of”. In this paper，an adaptive attention model is proposed, in which the encoder adopts the Faster R-CNN network to extract the salient features of images, the decoder LSTM network adapts a visual sentinel. At each time step, it can automatically decide when to rely on visual signals and when to just rely on the language model. Finally, the model is verified on Flickr30K and MS-COCO data sets, the experimental results show that the model effectively improves the quality of image captioning.

Key words: attention mechanism, Convolution Neural Network(CNN), Long Short-Term Memory Network(LSTM), image captioning