Dual-Stream Recurrent Neural Network for Video Captioning

doi:10.1109/TCSVT.2018.2867286

CORC > 计算技术研究所 > 中国科学院计算技术研究所 > 中国科学院计算技术研究所期刊论文 > 英文

	Dual-Stream Recurrent Neural Network for Video Captioning
	Xu, Ning 5; Liu, An-An 5; Wong, Yongkang 1; Zhang, Yongdong 2,3; Nie, Weizhi 5; Su, Yuting 5; Kankanhalli, Mohan 4
刊名	IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY
	2019-08-01
卷号	29 期号:8 页码:2482-2493
关键词	Video captioning hidden state fusion dual stream recurrent neural network attention module
ISSN号	1051-8215
DOI	10.1109/TCSVT.2018.2867286
英文摘要	Recent progress in using recurrent neural networks (RNNs) for video description has attracted an increasing interest, due to its capability to encode a sequence of frames for caption generation. While existing methods have studied various features (e.g., CNN, 3D CNN, and semantic attributes) for visual encoding, the representation and fusion of heterogeneous information from multi-modal spaces have not fully explored. Consider that different modalities are often asynchronous, frame-level multi-modal fusion (e.g., concatenation and linear fusion) will negatively influence each modality. In this paper, we propose a dual-stream RNN (DS-RNN) framework to jointly discover and integrate the hidden states of both visual and semantic streams for video caption generation. First, an encoding RNN is used for each stream to flexibly exploit the hidden states of respective modality. Specifically, we proposed an attentive multi-grained encoder module to enhance the local feature learning with global semantics feature. Then, a dual-stream decoder is deployed to integrate the asynchronous yet complementary sequential hidden states from both streams for caption generation. Extensive experiments on three benchmark datasets, namely, MSVD, MSR-VTT, and MPII-MD, show that DS-RNN achieves competitive performance against the state-of-the-art. Additional ablation studies were conducted on various variants of the proposed DS-RNN.
资助项目	National Natural Science Foundation of China[61772359] ; National Natural Science Foundation of China[61472275] ; National Natural Science Foundation of China[61525206] ; National Natural Science Foundation of China[61502337] ; National Key Research and Development Program of China[2017YFC0820600] ; National Defense Science and Technology Fund for Distinguished Young Scholars[2017-JCJQ-ZQ-022] ; National Research Foundation, Prime Minister's Office, Singapore, under its International Research Centre in Singapore Funding Initiative
WOS研究方向	Engineering
语种	英语
出版者	IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
WOS记录号	WOS:000480310500022
内容类型	期刊论文
源URL	[http://119.78.100.204/handle/2XEOYT63/4434]
专题	中国科学院计算技术研究所期刊论文_英文
通讯作者	Liu, An-An
作者单位	1.Natl Univ Singapore, Smart Syst Inst, Singapore 119077, Singapore 2.Univ Sci & Technol China, Sch Informat Sci & Technol, Hefei 230026, Anhui, Peoples R China 3.Chinese Acad Sci, Inst Comp Technol, Key Lab Intelligent Informat Proc, Beijing 100190, Peoples R China 4.Natl Univ Singapore, Sch Comp, Singapore 119077, Singapore 5.Tianjin Univ, Sch Elect & Informat Engn, Tianjin 300072, Peoples R China
推荐引用方式 GB/T 7714	Xu, Ning,Liu, An-An,Wong, Yongkang,et al. Dual-Stream Recurrent Neural Network for Video Captioning[J]. IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY,2019,29(8):2482-2493.
APA	Xu, Ning.,Liu, An-An.,Wong, Yongkang.,Zhang, Yongdong.,Nie, Weizhi.,...&Kankanhalli, Mohan.(2019).Dual-Stream Recurrent Neural Network for Video Captioning.IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY,29(8),2482-2493.
MLA	Xu, Ning,et al."Dual-Stream Recurrent Neural Network for Video Captioning".IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY 29.8(2019):2482-2493.