Statistical mechanics of continual learning: Variational principle and mean-field potential | |
Li, Chan; Huang, Zhenye1; Zou, Wenxuan; Huang, Haiping2 | |
刊名 | PHYSICAL REVIEW E |
2023 | |
卷号 | 108期号:1页码:14309 |
关键词 | NETWORKS |
ISSN号 | 2470-0045 |
DOI | 10.1103/PhysRevE.108.014309 |
英文摘要 | An obstacle to artificial general intelligence is set by continual learning of multiple tasks of a different nature. Recently, various heuristic tricks, both from machine learning and from neuroscience angles, were proposed, but they lack a unified theory foundation. Here, we focus on continual learning in single-layered and multilayered neural networks of binary weights. A variational Bayesian learning setting is thus proposed in which the neural networks are trained in a field-space, rather than a gradient-ill-defined discrete-weight space, and furthermore, weight uncertainty is naturally incorporated, and it modulates synaptic resources among tasks. From a physics perspective, we translate variational continual learning into a Franz-Parisi thermodynamic potential framework, where previous task knowledge serves as a prior probability and a reference as well. We thus interpret the continual learning of the binary perceptron in a teacher-student setting as a Franz-Parisi potential computation. The learning performance can then be analytically studied with mean-field order parameters, whose predictions coincide with numerical experiments using stochastic gradient descent methods. Based on the variational principle and Gaussian field approximation of internal preactivations in hidden layers, we also derive the learning algorithm considering weight uncertainty, which solves the continual learning with binary weights using multilayered neural networks, and performs better than the currently available metaplasticity algorithm in which binary synapses bear hidden continuous states and the synaptic plasticity is modulated by a heuristic regularization function. Our proposed principled frameworks also connect to elastic weight consolidation, weight-uncertainty modulated learning, and neuroscience-inspired metaplasticity, providing a theoretically grounded method for real-world multitask learning with deep networks. |
学科主题 | Physics |
语种 | 英语 |
内容类型 | 期刊论文 |
源URL | [http://ir.itp.ac.cn/handle/311006/28081] |
专题 | 理论物理研究所_理论物理所1978-2010年知识产出 |
作者单位 | 1.Sun Yat Sen Univ, Sch Phys, PMI Lab, Guangzhou 510275, Peoples R China 2.Chinese Acad Sci, CAS Key Lab Theoret Phys, Inst Theoret Phys, Beijing 100190, Peoples R China 3.Sun Yat sen Univ, Guangdong Prov Key Lab Magnetoelectr Phys & Device, Guangzhou 510275, Peoples R China |
推荐引用方式 GB/T 7714 | Li, Chan,Huang, Zhenye,Zou, Wenxuan,et al. Statistical mechanics of continual learning: Variational principle and mean-field potential[J]. PHYSICAL REVIEW E,2023,108(1):14309. |
APA | Li, Chan,Huang, Zhenye,Zou, Wenxuan,&Huang, Haiping.(2023).Statistical mechanics of continual learning: Variational principle and mean-field potential.PHYSICAL REVIEW E,108(1),14309. |
MLA | Li, Chan,et al."Statistical mechanics of continual learning: Variational principle and mean-field potential".PHYSICAL REVIEW E 108.1(2023):14309. |
个性服务 |
查看访问统计 |
相关权益政策 |
暂无数据 |
收藏/分享 |
除非特别说明,本系统中所有内容都受版权保护,并保留所有权利。
修改评论