Image Captioning Based on Adaptive Attention Model

Abstract

Abstract: Attention-based neural encoder-decoder frameworks have been widely adopted for image captioning. Most methods force visual attention to be active for every generated word. However, the decoder likely requires little to no visual information from the image to predict nonvisual words such as “the” and “of”. In this paper，an adaptive attention model is proposed, in which the encoder adopts the Faster R-CNN network to extract the salient features of images, the decoder LSTM network adapts a visual sentinel. At each time step, it can automatically decide when to rely on visual signals and when to just rely on the language model. Finally, the model is verified on Flickr30K and MS-COCO data sets, the experimental results show that the model effectively improves the quality of image captioning.

Key words: attention mechanism, Convolution Neural Network(CNN), Long Short-Term Memory Network(LSTM), image captioning

CLC Number:

TP391.41

HOU Xing-chen, WANG Jin. Image Captioning Based on Adaptive Attention Model[J]. Computer and Modernization, 2020, 0(06): 95-.

References

［1］ PLUMMER B A, WANG L, CERVANTES C M， et al. Flickr30k entities:Collecting region-to-phrase correspondences for richer image-to-sentence models［J］. International Journal of Computer Vision, 2017,123(1):74-93.
［2］ VINYALS O, TOSHEV A, BENGIO S， et al. Show and tell: A neural image caption generator［C］// Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2015:3156-3164.
［3］ XIONG L C, PARIKH D, SOCHER R. Knowing when to look: Adaptive attention via a visual sentinel for image captioning［J］. Computer Vision and Pattern Recognition, arXiv：1612.01887， 2017.
［4］ YU D, FU J, MEI T, et al. Multi-level attention networks for visual question answering［C］ // 2017 IEEE Conference on Computer Vision and Pattern Recognition. 2017:4187-4195.
［5］ XU K， BA J， KIROS R， et al. Show， attend and tell： Neural image caption generation with visual attention［J］. Machine Learning, arXiv：1502.03044， 2015.
［6］ YANG Z, YUAN Y, WU Y, et al. Review networks for caption generation［C］// Advances in Neural Information Processing Systems. 2016:2361-2369.
［7］ REN S， HE K， GIRSHICK R， et al. Faster R-CNN： Towards real-time object detection with region proposal networks［J］. IEEE Transactions on Pattern Analysis & Machine Intelligence， 2017，39（6）：1137-1149.
［8］ HE K, ZHANG X, REN S, et al. Delving deepinto rectifiers: Surpassing human-level performance on image net classification［C］// Proceedings of IEEE International Conference on Computer Vision. 2015:1026-1034.
［9］ FARHADI A, HEJRATI M, SADEGHI M A, et al. Every picture tells a story: Generating sentences from images［J］. Lecture Notes in Computer Science, 2010,21(10):15-29.
［10］LI S, KULKARNI G, BERG T, et al. Composing simple image descriptions using web-scale n-grams［C］// ACL Conference on Computational Natural Language Learning. 2011:220-228.
［11］KULKARNI G, PREMRAJ V, ORDONEZ V, et al. Baby talk: Understanding and generating simple image descriptions［C］// IEEE Conference on Computer Vision and Pattern Recognition. 2011:1601-1608.
［12］CHO K, VAN MERRIENBOER B, GULCEHRE C, et al.Learning phrase representations using RNN encoder-decoder for statistical machine translation［J］. Computation and Language, arXiv:1406.1078, 2014.
［13］SUTSKEVER I, VINYALS O, LE Q V. Sequence to sequence learning with neural networks［C］// Advances in Neural Information Processing Systems. 2014:3104-3112.
［14］BAHDANAU D, CHO K, BENGIO Y. Neural machine translation by jointly learning to align and translate［J］. Computation and Language, arXiv:1409.0473, 2014.
［15］KIROS R, ALAKHUTDINOV R, ZEMEL R S. Multimodal neural language models［C］// ICML. 2014:595-603.
［16］WU Q, SHEN C, LIU L, et al. What value do explicit high level concepts have invision to language problems?［J］. Computer Vision and Pattern Recognition, arXiv:1506.01144, 2015.
［17］YOU Q, JIN H, WANG Z, et al. Image captioning with semantic attention［C］// 2016 IEEE Conference on Computer Vision and Pattern Recognition. 2016:4651-4659.
［18］YAO T, PAN Y, LI Y, et al. Boosting image captioning with attributes［J］. Computer Vision and Pattern Recognition, arXiv:1611.01646,2015.
［19］KRISHNA R, ZHU Y, GROTH O, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations［J］. Computer Vision and Pattern Recognition, arXiv:1602.07332,2016.
［20］LIN T Y， MAIRE M， BELONGIE S， et al. Microsoft COCO： Common objects in context［J］. Computer Vision and Pattern Recognition, arXiv：1405.0312，2014.
［21］PAPINENI K, ROUKOS S, WARD T, et al. BLEU: A method for automatic evaluation of machine translation［C］// Proceedings of the 40th Annual Meeting on Association for Computational Linguistics. 2002:311-318.
［22］BANERJEE S, LAVIE A. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments［C］// ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization. 2005:65-72.
［23］FLICK C. ROUGE: A package for automatic evaluation of summaries［C］// The Workshop on Text Summarization Branches Out. 2004:25-26.
［24］VEDANTAM R, ZITNICK C L, PARIKH D. CIDEr: Consensus-based image description evaluation［C］// 2015 IEEE Conference on Computer Vision and Pattern Recognition. 2015:4566-4575.

[1]	HE Sida, CHEN Pinghua. Intent-based Lightweight Self-Attention Network for Sequential Recommendation [J]. Computer and Modernization, 2024, 0(12): 1-9.
[2]	ZHAO Chenyang, XUE Tao, LIU Junhua. Fashion Clothing Pattern Generation Based on Improved Stable Diffusion [J]. Computer and Modernization, 2024, 0(12): 15-23.
[3]	HUANG Tingpei1, MA Lubiao1, LI Shibao2, LIU Jianhang1. Gesture Recognition Method Based on WiFi and Prototypical Network [J]. Computer and Modernization, 2024, 0(12): 34-39.
[4]	ZHANG Xiaodong1, BAI Guangzhi1, LI Min1, LI Haoyang2. Oil and Gas Well Production Prediction Model Based on Empirical Wavelet Transform [J]. Computer and Modernization, 2024, 0(12): 53-58.
[5]	WANG Yanyuan, MAO Zhengchong. Detection and Recognition Algorithms for Chinese and English Scene Text Images [J]. Computer and Modernization, 2024, 0(12): 84-90.
[6]	LI Junchao1, YOU Fei1, ZHANG Chao2, SU Lele2, GONG Yan2 . BiLSTM-Attention Prediction Model and Error Analysis #br# Based on Novel Multi-objective Coati Optimization Algorithm [J]. Computer and Modernization, 2024, 0(11): 70-76.
[7]	ZHANG Yu1, 2, LI Jing1, 2, MA Ming1, 2, WANG Zhongxiang1, 2, SUN Yan1, 2. YOLOLW: A Novel Lightweight Object Detection Model [J]. Computer and Modernization, 2024, 0(11): 91-98.
[8]	QI Xian, LIU Daming, CHANG Jiaxin. Multi-view 3D Reconstruction Based on Improved Self-attention Mechanism [J]. Computer and Modernization, 2024, 0(11): 106-112.
[9]	YANG Jun1, HU Wei1, ZHU Wenfu2. Visual SLAM Loop Closure Detection Algorithm Based on Improved MobileNetV3 [J]. Computer and Modernization, 2024, 0(10): 21-26.
[10]	WEI Xuecheng1, JIANG Lingyun1, LI Yan2, HE Fei2. Improved Roadside Monocular View Small Target Detection Algorithm Based on YOLOv5 [J]. Computer and Modernization, 2024, 0(10): 27-34.
[11]	DU Mengjun1, LI Ang1, TONG Jun1, QIAN Jin1, KANG Kai1, WANG Ruoding1, JIN Wenxing2. Power Information Data Fusion Model Based on Improved Extreme Learning Algorithm [J]. Computer and Modernization, 2024, 0(10): 61-64.
[12]	YANG Shijun1, DI Guangyi1, GAO Jun1, CHEN Jianfei1, WANG Yaokun1, JI Xiaohan2. Sentiment Consistency Detection Based on Cross Modal Attention Fusion and#br# Information Perception [J]. Computer and Modernization, 2024, 0(10): 113-119.
[13]	HOU Congying, YANG Wengqing, WANG Zhao, CHENG Cong. Speech Enhancement Based on Time-frequency Self-attention Residual Temporal#br# Convolutional Networks [J]. Computer and Modernization, 2024, 0(09): 20-24.
[14]	ZHANG Ze1, ZHANG Jianquan2, 3, ZHOU Guopeng2, 3. Camera Module Defect Detection Based on Improved YOLOv8s [J]. Computer and Modernization, 2024, 0(09): 107-113.
[15]	ZHENG Shangpo1, CHEN Defu1, LI Jianli2, LIN Guoxian2, WANG Xingping3. Pedestrian Tracking Algorithm Based on Improved YOLOv5s and DeepSORT [J]. Computer and Modernization, 2024, 0(08): 54-58.