Neural Encoding for Human Visual Cortex With Deep Neural Networks Learning "What" and "Where"

doi:10.1109/TCDS.2020.3007761

CORC > 自动化研究所 > 中国科学院自动化研究所 > 类脑智能研究中心 > 神经计算及脑机交互

	Neural Encoding for Human Visual Cortex With Deep Neural Networks Learning "What" and "Where"
	Wang, Haibao 2,3; Huang, Lijie 2,3; Du, Changde 2,3; Li, Dan 2,3; Wang, Bo 2,3; He, Huiguang 1,2,3
刊名	IEEE TRANSACTIONS ON COGNITIVE AND DEVELOPMENTAL SYSTEMS
	2021-12-01
卷号	13 期号:4 页码:827-840
关键词	Visualization Feature extraction Encoding Brain modeling Biological neural networks Sociology Statistics Deep neural network (DNN) neural encoding regularization "what" and "where"
ISSN号	2379-8920
DOI	10.1109/TCDS.2020.3007761
通讯作者	He, Huiguang(huiguang.he@ia.ac.cn)
英文摘要	Neural encoding, a crucial aspect to understand the human brain information processing system, aims to establish a quantitative relationship between the stimuli and the evoked brain activities. In the field of visual neuroscience, with the ability to explain how neurons in the primary visual cortex work, population receptive field (pRF) models have enjoyed high popularity and made reliable progress in recent years. However, existing models rely on either the inflexible prior assumptions about pRF or the clumsy parameter estimation methods, severely limiting the expressiveness and interpretability. In this article, we propose a novel neural encoding framework by learning "what" and "where" with deep neural networks. It involves two separate aspects: 1) the spatial characteristic ("where") and 2) feature selection ("what") of neuron populations in the visual cortex. Specifically, our approach first encodes visual stimuli into hierarchically intermediate features through a pretrained deep neural network (DNN), then converts DNN features into refined features with the channel attention and spatial receptive field (RF) to learn "where", and finally regresses refined features simultaneously onto voxel activities to learn "what". The sparsity regularization and smoothness regularization are adopted in our modeling approach so that the crucial RF can be estimated automatically without prior assumptions about shapes. Furthermore, an attempt is made to extend the voxel-wise modeling approach to multi-voxel joint encoding models, and we show that it is conducive to rescuing voxels with poor signal-to-noise characteristics. Extensive empirical results demonstrate that the method developed herein provides an effective strategy to establish neural encoding for the human visual cortex, with the weaker prior constraints but the higher encoding performance.
资助项目	National Natural Science Foundation of China[61976209] ; National Natural Science Foundation of China[61906188] ; CAS International Collaboration Key Project[173211KYSB20190024] ; Strategic Priority Research Program of CAS[XDB32040000]
WOS关键词	RECEPTIVE-FIELD ; REPRESENTATION ; ATTENTION ; IMAGES ; MODELS
WOS研究方向	Computer Science ; Robotics ; Neurosciences & Neurology
语种	英语
出版者	IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
WOS记录号	WOS:000728925200012
资助机构	National Natural Science Foundation of China ; CAS International Collaboration Key Project ; Strategic Priority Research Program of CAS
内容类型	期刊论文
源URL	[http://ir.ia.ac.cn/handle/173211/46767]
专题	类脑智能研究中心_神经计算及脑机交互
通讯作者	He, Huiguang
作者单位	1.Chinese Acad Sci, Ctr Excellence Brain Sci & Intelligence Technol, Beijing 100190, Peoples R China 2.Univ Chinese Acad Sci, Sch Artificial Intelligence, Beijing 100049, Peoples R China 3.Chinese Acad Sci, Res Ctr Brain Inspired Intelligence, Inst Automat, Natl Lab Pattern Recognit, Beijing 100190, Peoples R China
推荐引用方式 GB/T 7714	Wang, Haibao,Huang, Lijie,Du, Changde,et al. Neural Encoding for Human Visual Cortex With Deep Neural Networks Learning "What" and "Where"[J]. IEEE TRANSACTIONS ON COGNITIVE AND DEVELOPMENTAL SYSTEMS,2021,13(4):827-840.
APA	Wang, Haibao,Huang, Lijie,Du, Changde,Li, Dan,Wang, Bo,&He, Huiguang.(2021).Neural Encoding for Human Visual Cortex With Deep Neural Networks Learning "What" and "Where".IEEE TRANSACTIONS ON COGNITIVE AND DEVELOPMENTAL SYSTEMS,13(4),827-840.
MLA	Wang, Haibao,et al."Neural Encoding for Human Visual Cortex With Deep Neural Networks Learning "What" and "Where"".IEEE TRANSACTIONS ON COGNITIVE AND DEVELOPMENTAL SYSTEMS 13.4(2021):827-840.