Image Captioning with Word Gate and Adaptive Self-Critical Learning

doi:10.3390/app8060909

	Image Captioning with Word Gate and Adaptive Self-Critical Learning
	Zhu, Xinxin 1,2; Li, Lixiang 1,2; Liu, Jing3 ; Guo, Longteng 3; Fang, Zhiwei3 ; Peng, Haipeng 1,2; Niu, Xinxin 1,2
刊名	APPLIED SCIENCES-BASEL
	2018-06-01
卷号	8 期号:6 页码:13
关键词	image caption image understanding deep learning computer vision
ISSN号	2076-3417
DOI	10.3390/app8060909
通讯作者	Li, Lixiang(li_lixiang2006@163.com)
英文摘要	Although the policy-gradient methods for reinforcement learning have shown significant improvement in image captioning, how to achieve high performance during the reinforcement optimizing process is still not a simple task. There are at least two difficulties: (1) The large size of vocabulary leads to a large action space, which makes it difficult for the model to accurately predict the current word. (2) The large variance of gradient estimation in reinforcement learning usually causes severe instabilities in the training process. In this paper, we propose two innovations to boost the performance of self-critical sequence training (SCST). First, we modify the standard long short-term memory (LSTM)based decoder by introducing a gate function to reduce the search scope of the vocabulary for any given image, which is termed the word gate decoder. Second, instead of only considering current maximum actions greedily, we propose a stabilized gradient estimation method whose gradient variance is controlled by the difference between the sampling reward from the current model and the expectation of the historical reward. We conducted extensive experiments, and results showed that our method could accelerate the training process and increase the prediction accuracy. Our method was validated on MS COCO datasets and yielded state-of-the-art performance.
资助项目	National Key Research and Development Program of China[2016YFB0800602] ; National Natural Science Foundation of China[61771071] ; National Natural Science Foundation of China[61573067]
WOS研究方向	Chemistry ; Materials Science ; Physics
语种	英语
出版者	MDPI
WOS记录号	WOS:000436488000068
资助机构	National Key Research and Development Program of China ; National Natural Science Foundation of China
内容类型	期刊论文
源URL	[http://ir.ia.ac.cn/handle/173211/26324]
专题	中国科学院自动化研究所
通讯作者	Li, Lixiang
作者单位	1.Beijing Univ Posts & Telecommun, Informat Secur Ctr, State Key Lab Networking & Switching Technol, Beijing 100876, Peoples R China 2.Beijing Univ Posts & Telecommun, Natl Engn Lab Disaster Backup & Recovery, Beijing 100876, Peoples R China 3.Chinese Acad Sci, Inst Automat, Natl Lab Pattern Recognit, Beijing 100190, Peoples R China
推荐引用方式 GB/T 7714	Zhu, Xinxin,Li, Lixiang,Liu, Jing,et al. Image Captioning with Word Gate and Adaptive Self-Critical Learning[J]. APPLIED SCIENCES-BASEL,2018,8(6):13.
APA	Zhu, Xinxin.,Li, Lixiang.,Liu, Jing.,Guo, Longteng.,Fang, Zhiwei.,...&Niu, Xinxin.(2018).Image Captioning with Word Gate and Adaptive Self-Critical Learning.APPLIED SCIENCES-BASEL,8(6),13.
MLA	Zhu, Xinxin,et al."Image Captioning with Word Gate and Adaptive Self-Critical Learning".APPLIED SCIENCES-BASEL 8.6(2018):13.