Structured Neural Decoding With Multitask Transfer Learning of Deep Neural Network Representations

doi:10.1109/TNNLS.2020.3028167

CORC > 自动化研究所 > 中国科学院自动化研究所 > 类脑智能研究中心 > 神经计算及脑机交互

	Structured Neural Decoding With Multitask Transfer Learning of Deep Neural Network Representations
	Du, Changde 1,2,3; Du, Changying 5; Huang, Lijie 3; Wang, Haibao 2,3; He, Huiguang 2,3,4
刊名	IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS
	2022-02-01
卷号	33 期号:2 页码:600-614
关键词	Decoding Image reconstruction Functional magnetic resonance imaging Visualization Task analysis Brain Correlation Deep neural network (DNN) functional magnetic resonance imaging (fMRI) image reconstruction multioutput regression neural decoding
ISSN号	2162-237X
DOI	10.1109/TNNLS.2020.3028167
通讯作者	He, Huiguang(huiguang.he@ia.ac.cn)
英文摘要	The reconstruction of visual information from human brain activity is a very important research topic in brain decoding. Existing methods ignore the structural information underlying the brain activities and the visual features, which severely limits their performance and interpretability. Here, we propose a hierarchically structured neural decoding framework by using multitask transfer learning of deep neural network (DNN) representations and a matrix-variate Gaussian prior. Our framework consists of two stages, Voxel2Unit and Unit2Pixel. In Voxel2Unit, we decode the functional magnetic resonance imaging (fMRI) data to the intermediate features of a pretrained convolutional neural network (CNN). In Unit2Pixel, we further invert the predicted CNN features back to the visual images. Matrix-variate Gaussian prior allows us to take into account the structures between feature dimensions and between regression tasks, which are useful for improving decoding effectiveness and interpretability. This is in contrast with the existing single-output regression models that usually ignore these structures. We conduct extensive experiments on two real-world fMRI data sets, and the results show that our method can predict CNN features more accurately and reconstruct the perceived natural images and faces with higher quality.
资助项目	National Natural Science Foundation of China[61976209] ; National Natural Science Foundation of China[62020106015] ; National Natural Science Foundation of China[61906188] ; National Natural Science Foundation of China[61602449] ; Chinese Academy of Sciences (CAS) International Collaboration Key Project[173211KYSB20190024] ; CAS[XDB32040000]
WOS关键词	NATURAL IMAGES ; BRAIN ; RECONSTRUCTION ; FACES
WOS研究方向	Computer Science ; Engineering
语种	英语
出版者	IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
WOS记录号	WOS:000752016400015
资助机构	National Natural Science Foundation of China ; Chinese Academy of Sciences (CAS) International Collaboration Key Project ; CAS
内容类型	期刊论文
源URL	[http://ir.ia.ac.cn/handle/173211/47361]
专题	类脑智能研究中心_神经计算及脑机交互
通讯作者	He, Huiguang
作者单位	1.Huawei Cloud BU EI Innovat Lab, Beijing 100085, Peoples R China 2.Univ Chinese Acad Sci, Sch Artificial Intelligence, Beijing 100049, Peoples R China 3.Chinese Acad Sci, Res Ctr Brain Inspired Intelligence, Natl Lab Pattern Recognit, Inst Automat, Beijing 100190, Peoples R China 4.Chinese Acad Sci, Ctr Excellence Brain Sci & Intelligence Technol, Beijing 100190, Peoples R China 5.Huawei Noahs Ark Lab, Beijing 100085, Peoples R China
推荐引用方式 GB/T 7714	Du, Changde,Du, Changying,Huang, Lijie,et al. Structured Neural Decoding With Multitask Transfer Learning of Deep Neural Network Representations[J]. IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS,2022,33(2):600-614.
APA	Du, Changde,Du, Changying,Huang, Lijie,Wang, Haibao,&He, Huiguang.(2022).Structured Neural Decoding With Multitask Transfer Learning of Deep Neural Network Representations.IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS,33(2),600-614.
MLA	Du, Changde,et al."Structured Neural Decoding With Multitask Transfer Learning of Deep Neural Network Representations".IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS 33.2(2022):600-614.