基于自适应注意模型的图像描述

摘要/Abstract

摘要： 结合注意力机制的编解码框架模型已经被广泛地应用在图像描述任务中。大多数方法都强制对生成的每个单词进行主动的视觉注意，然而，解码器很可能不需要关注图像中的任何视觉信息就生成非视觉单词，比如“the”和“of”。本文提出一种自适应注意力模型，编码端采用Faster R-CNN网络提取图像中的显著特征，解码端LSTM网络中引入一个视觉监督信号。在每个时间步长，它可以自动地决定何时依赖于视觉信号，何时仅依赖于语言模型。最后在Flickr30K和MS-COCO数据集进行验证，实验结果表明该模型有效地提升了描述语句的质量。

关键词: 注意力机制, 卷积神经网络, 长短时记忆网络(LSTM), 图像描述

Abstract: Attention-based neural encoder-decoder frameworks have been widely adopted for image captioning. Most methods force visual attention to be active for every generated word. However, the decoder likely requires little to no visual information from the image to predict nonvisual words such as “the” and “of”. In this paper，an adaptive attention model is proposed, in which the encoder adopts the Faster R-CNN network to extract the salient features of images, the decoder LSTM network adapts a visual sentinel. At each time step, it can automatically decide when to rely on visual signals and when to just rely on the language model. Finally, the model is verified on Flickr30K and MS-COCO data sets, the experimental results show that the model effectively improves the quality of image captioning.

Key words: attention mechanism, Convolution Neural Network(CNN), Long Short-Term Memory Network(LSTM), image captioning

中图分类号:

TP391.41

侯星晨, 王锦. 基于自适应注意模型的图像描述[J]. 计算机与现代化, 2020, 0(06): 95-.

HOU Xing-chen, WANG Jin. Image Captioning Based on Adaptive Attention Model[J]. Computer and Modernization, 2020, 0(06): 95-.

参考文献［24］

［1］	PLUMMER B A, WANG L, CERVANTES C M， et al. Flickr30k entities:Collecting region-to-phrase correspondences for richer image-to-sentence models［J］. International Journal of Computer Vision, 2017,123(1):74-93.
［2］	VINYALS O, TOSHEV A, BENGIO S， et al. Show and tell: A neural image caption generator［C］// Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2015:3156-3164.
［3］	XIONG L C, PARIKH D, SOCHER R. Knowing when to look: Adaptive attention via a visual sentinel for image captioning［J］. Computer Vision and Pattern Recognition, arXiv：1612.01887， 2017.
［4］	YU D, FU J, MEI T, et al. Multi-level attention networks for visual question answering［C］ // 2017 IEEE Conference on Computer Vision and Pattern Recognition. 2017:4187-4195.
［5］	XU K， BA J， KIROS R， et al. Show， attend and tell： Neural image caption generation with visual attention［J］. Machine Learning, arXiv：1502.03044， 2015.
［6］	YANG Z, YUAN Y, WU Y, et al. Review networks for caption generation［C］// Advances in Neural Information Processing Systems. 2016:2361-2369.
［7］	REN S， HE K， GIRSHICK R， et al. Faster R-CNN： Towards real-time object detection with region proposal networks［J］. IEEE Transactions on Pattern Analysis & Machine Intelligence， 2017，39（6）：1137-1149.
［8］	HE K, ZHANG X, REN S, et al. Delving deepinto rectifiers: Surpassing human-level performance on image net classification［C］// Proceedings of IEEE International Conference on Computer Vision. 2015:1026-1034.
［9］	FARHADI A, HEJRATI M, SADEGHI M A, et al. Every picture tells a story: Generating sentences from images［J］. Lecture Notes in Computer Science, 2010,21(10):15-29.
［10］	LI S, KULKARNI G, BERG T, et al. Composing simple image descriptions using web-scale n-grams［C］// ACL Conference on Computational Natural Language Learning. 2011:220-228.
［11］	KULKARNI G, PREMRAJ V, ORDONEZ V, et al. Baby talk: Understanding and generating simple image descriptions［C］// IEEE Conference on Computer Vision and Pattern Recognition. 2011:1601-1608.
［12］	CHO K, VAN MERRIENBOER B, GULCEHRE C, et al.Learning phrase representations using RNN encoder-decoder for statistical machine translation［J］. Computation and Language, arXiv:1406.1078, 2014.
［13］	SUTSKEVER I, VINYALS O, LE Q V. Sequence to sequence learning with neural networks［C］// Advances in Neural Information Processing Systems. 2014:3104-3112.
［14］	BAHDANAU D, CHO K, BENGIO Y. Neural machine translation by jointly learning to align and translate［J］. Computation and Language, arXiv:1409.0473, 2014.
［15］	KIROS R, ALAKHUTDINOV R, ZEMEL R S. Multimodal neural language models［C］// ICML. 2014:595-603.
［16］	WU Q, SHEN C, LIU L, et al. What value do explicit high level concepts have invision to language problems?［J］. Computer Vision and Pattern Recognition, arXiv:1506.01144, 2015.
［17］	YOU Q, JIN H, WANG Z, et al. Image captioning with semantic attention［C］// 2016 IEEE Conference on Computer Vision and Pattern Recognition. 2016:4651-4659.
［18］	YAO T, PAN Y, LI Y, et al. Boosting image captioning with attributes［J］. Computer Vision and Pattern Recognition, arXiv:1611.01646,2015.
［19］	KRISHNA R, ZHU Y, GROTH O, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations［J］. Computer Vision and Pattern Recognition, arXiv:1602.07332,2016.
［20］	LIN T Y， MAIRE M， BELONGIE S， et al. Microsoft COCO： Common objects in context［J］. Computer Vision and Pattern Recognition, arXiv：1405.0312，2014.
［21］	PAPINENI K, ROUKOS S, WARD T, et al. BLEU: A method for automatic evaluation of machine translation［C］// Proceedings of the 40th Annual Meeting on Association for Computational Linguistics. 2002:311-318.
［22］	BANERJEE S, LAVIE A. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments［C］// ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization. 2005:65-72.
［23］	FLICK C. ROUGE: A package for automatic evaluation of summaries［C］// The Workshop on Text Summarization Branches Out. 2004:25-26.
［24］	VEDANTAM R, ZITNICK C L, PARIKH D. CIDEr: Consensus-based image description evaluation［C］// 2015 IEEE Conference on Computer Vision and Pattern Recognition. 2015:4566-4575.

[1]	王秋忆, 周浩, 郑婷婷. 改进RetinaNet的电力设备目标检测方法[J]. 计算机与现代化, 2024, 0(01): 47-52.
[2]	林启钊, 彭志平, 郭棉, 崔得龙. 基于双向多步预测的炉管温度场重构方法[J]. 计算机与现代化, 2024, 0(01): 53-58.
[3]	李亚平, 王军防, 余红梅, 窦一民, 肖媛, 田继林. Regformer：基于稀疏注意力的输油管道水力压降预测方法[J]. 计算机与现代化, 2024, 0(01): 59-66.
[4]	周成诚, 曾庆军, 杨康, 胡家铭, 韩春伟. 基于高效通道注意力模块的运动想象脑电识别[J]. 计算机与现代化, 2023, 0(12): 19-23.
[5]	邱凯星, 冯广. 基于双重特征注意力的多标签图像分类模型[J]. 计算机与现代化, 2023, 0(12): 41-47.
[6]	张浩洋, 尹梓名, 乐珺怡, 沈达聪, 束翌俊, 杨自逸, 孔祥勇, 龚伟. 3D-SPRNet: 一种基于并行解码器和双注意力机制的胆囊癌分割模型[J]. 计算机与现代化, 2023, 0(12): 59-66.
[7]	张伯泉, 麦海鹏, 陈嘉敏, 逄锦聚. 基于高灰度值注意力机制的脑白质高信号分割[J]. 计算机与现代化, 2023, 0(12): 67-75.
[8]	宁娟, 周庆华, 曾小为. 改进YOLOv7算法在西林瓶轧盖缺陷检测中的应用[J]. 计算机与现代化, 2023, 0(12): 82-86.
[9]	王宇航, 董宝良, 公超, 尚真真, 姚康宁. 基于意图识别的空中群目标动态威胁评估[J]. 计算机与现代化, 2023, 0(12): 100-104.
[10]	罗明杰, 冯开平. 基于沙漏结构与注意力机制的轻量级人脸表情识别方法[J]. 计算机与现代化, 2023, 0(11): 89-94.
[11]	张嘉琪, 徐啟蕾. 基于NAM-YOLO网络的苹果缺陷检测算法[J]. 计算机与现代化, 2023, 0(10): 53-58.
[12]	叶思佳, 魏延, 杜韩宇, 邓金枝. 结合注意力机制的HRNet图像语义分割算法[J]. 计算机与现代化, 2023, 0(10): 65-69.
[13]	刘付琪, 张达, 宋建华, 王海东. 基于CNN-BiLSTM的液压系统故障诊断[J]. 计算机与现代化, 2023, 0(09): 10-19.
[14]	吴甜, 刘海华, 童顺延. 基于深度反馈的卷积神经网络的图像分类[J]. 计算机与现代化, 2023, 0(09): 82-86.
[15]	陈嘉敏, 张伯泉, 麦海鹏. 基于特征融合的海马体分割[J]. 计算机与现代化, 2023, 0(08): 1-6.