Recall What You See Continually Using GridLSTM in Image Captioning

doi:10.1109/TMM.2019.2931815

CORC > 自动化研究所 > 中国科学院自动化研究所 > 模式识别国家重点实验室 > 图像与视频分析团队

	Recall What You See Continually Using GridLSTM in Image Captioning
	Wu, Lingxiang 1; Xu, Min 1; Wang, Jinqiao 2; Perry, Stuart 3
刊名	IEEE TRANSACTIONS ON MULTIMEDIA
	2020-03-01
卷号	22 期号:3 页码:808-818
关键词	Visualization Decoding Task analysis Neural networks Training Computational modeling Logic gates Image captioning GridLSTM recurrent neural network
ISSN号	1520-9210
DOI	10.1109/TMM.2019.2931815
通讯作者	Xu, Min(Min.Xu@uts.edu.au)
英文摘要	The goal of image captioning is to automatically describe an image with a sentence, and the task has attracted research attention from both the computer vision and natural-language processing research communities. The existing encoder-decoder model and its variants, which are the most popular models for image captioning, use the image features in three ways: first, they inject the encoded image features into the decoder only once at the initial step, which does not enable the rich image content to be explored sufficiently while gradually generating a text caption; second, they concatenate the encoded image features with text as extra inputs at every step, which introduces unnecessary noise; and, third, they using an attention mechanism, which increases the computational complexity due to the introduction of extra neural nets to identify the attention regions. Different from the existing methods, in this paper, we propose a novel network, Recall Network, for generating captions that are consistent with the images. The recall network selectively involves the visual features by using a GridLSTM and, thus, is able to recall image contents while generating each word. By importing the visual information as the latent memory along the depth dimension LSTM, the decoder is able to admit the visual features dynamically through the inherent LSTM structure without adding any extra neural nets or parameters. The Recall Network efficiently prevents the decoder from deviating from the original image content. To verify the efficiency of our model, we conducted exhaustive experiments on full and dense image captioning. The experimental results clearly demonstrate that our recall network outperforms the conventional encoder-decoder model by a large margin and that it performs comparably to the state-of-the-art methods.
WOS关键词	CLASSIFICATION ; ATTENTION
WOS研究方向	Computer Science ; Telecommunications
语种	英语
出版者	IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
WOS记录号	WOS:000519576700019
内容类型	期刊论文
源URL	[http://ir.ia.ac.cn/handle/173211/38639]
专题	自动化研究所_模式识别国家重点实验室_图像与视频分析团队
通讯作者	Xu, Min
作者单位	1.Univ Technol Sydney, Global Big Data Technol Ctr, Sch Elect & Data Engn, Ultimo, NSW 2007, Australia 2.Chinese Acad Sci, Inst Automat, Natl Lab Pattern Recognit, Beijing 100190, Peoples R China 3.Univ Technol Sydney, Sch Elect & Data Engn, Perceptual Imaging Lab, Ultimo, NSW 2007, Australia
推荐引用方式 GB/T 7714	Wu, Lingxiang,Xu, Min,Wang, Jinqiao,et al. Recall What You See Continually Using GridLSTM in Image Captioning[J]. IEEE TRANSACTIONS ON MULTIMEDIA,2020,22(3):808-818.
APA	Wu, Lingxiang,Xu, Min,Wang, Jinqiao,&Perry, Stuart.(2020).Recall What You See Continually Using GridLSTM in Image Captioning.IEEE TRANSACTIONS ON MULTIMEDIA,22(3),808-818.
MLA	Wu, Lingxiang,et al."Recall What You See Continually Using GridLSTM in Image Captioning".IEEE TRANSACTIONS ON MULTIMEDIA 22.3(2020):808-818.