Know More Say Less: Image Captioning Based on Scene Graphs

doi:10.1109/TMM.2019.2896516

CORC > 计算技术研究所 > 中国科学院计算技术研究所 > 中国科学院计算技术研究所期刊论文 > 英文

	Know More Say Less: Image Captioning Based on Scene Graphs
	Li, Xiangyang 1,2; Jiang, Shuqiang 1,2
刊名	IEEE TRANSACTIONS ON MULTIMEDIA
	2019-08-01
卷号	21 期号:8 页码:2117-2130
关键词	Image captioning scene graph relationship long short-term network attention mechanism vision-language
ISSN号	1520-9210
DOI	10.1109/TMM.2019.2896516
英文摘要	Automatically describing the content of an image has been attracting considerable research attention in the multimedia field. To represent the content of an image, many approaches directly utilize convolutional neural networks (CNNs) to extract visual representations, which are fed into recurrent neural networks to generate natural language. Recently, some approaches have detected semantic concepts from images and then encoded them into high-level representations. Although substantial progress has been achieved, most of the previous methods treat entities in images individually, thus lacking structured information that provides important cues for image captioning. In this paper, we propose a framework based on scene graphs for image captioning. Scene graphs contain abundant structured information because they not only depict object entities in images but also present pairwise relationships. To leverage both visual features and semantic knowledge in structured scene graphs, we extract CNN features from the bounding box offsets of object entities for visual representations, and extract semantic relationship features from triples (e.g., man riding bike) for semantic representations. After obtaining these features, we introduce a hierarchical-attention-based module to learn discriminative features for word generation at each time step. The experimental results on benchmark datasets demonstrate the superiority of our method compared with several state-of-the-art methods.
资助项目	National Natural Science Foundation of China[61532018] ; Beijing Natural Science Foundation[L182054] ; National Program for Special Support of Eminent Professionals ; National Program for Support of Top-notch Young Professionals
WOS研究方向	Computer Science ; Telecommunications
语种	英语
出版者	IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
WOS记录号	WOS:000476809700018
内容类型	期刊论文
源URL	[http://119.78.100.204/handle/2XEOYT63/4478]
专题	中国科学院计算技术研究所期刊论文_英文
通讯作者	Jiang, Shuqiang
作者单位	1.Chinese Acad Sci, Inst Comp Technol, Key Lab Intelligent Informat Proc, Beijing 100190, Peoples R China 2.Univ Chinese Acad Sci, Beijing 100049, Peoples R China
推荐引用方式 GB/T 7714	Li, Xiangyang,Jiang, Shuqiang. Know More Say Less: Image Captioning Based on Scene Graphs[J]. IEEE TRANSACTIONS ON MULTIMEDIA,2019,21(8):2117-2130.
APA	Li, Xiangyang,&Jiang, Shuqiang.(2019).Know More Say Less: Image Captioning Based on Scene Graphs.IEEE TRANSACTIONS ON MULTIMEDIA,21(8),2117-2130.
MLA	Li, Xiangyang,et al."Know More Say Less: Image Captioning Based on Scene Graphs".IEEE TRANSACTIONS ON MULTIMEDIA 21.8(2019):2117-2130.