Theory of the Frequency Principle for General Deep Neural Networks
Year: 2021
Author: Tao Luo, Zheng Ma, Zhi-Qin John Xu, Yaoyu Zhang
CSIAM Transactions on Applied Mathematics, Vol. 2 (2021), Iss. 3 : pp. 484–507
Abstract
Along with fruitful applications of Deep Neural Networks (DNNs) to realistic problems, recently, empirical studies reported a universal phenomenon of Frequency Principle (F-Principle), that is, a DNN tends to learn a target function from low to high frequencies during the training. The F-Principle has been very useful in providing both qualitative and quantitative understandings of DNNs. In this paper, we rigorously investigate the F-Principle for the training dynamics of a general DNN at three stages: initial stage, intermediate stage, and final stage. For each stage, a theorem is provided in terms of proper quantities characterizing the F-Principle. Our results are general in the sense that they work for multilayer networks with general activation functions, population densities of data, and a large class of loss functions. Our work lays a theoretical foundation of the F-Principle for a better understanding of the training process of DNNs.
You do not have full access to this article.
Already a Subscriber? Sign in as an individual or via your institution
Journal Article Details
Publisher Name: Global Science Press
Language: English
DOI: https://doi.org/10.4208/csiam-am.SO-2020-0005
CSIAM Transactions on Applied Mathematics, Vol. 2 (2021), Iss. 3 : pp. 484–507
Published online: 2021-01
AMS Subject Headings: Global Science Press
Copyright: COPYRIGHT: © Global Science Press
Pages: 24
Keywords: Frequency principle Deep Neural Networks dynamical system training process.
Author Details
Tao Luo Email
Zheng Ma Email
Zhi-Qin John Xu Email
Yaoyu Zhang Email
-
Spectral-DP: Differentially Private Deep Learning through Spectral Perturbation and Filtering
Feng, Ce | Xu, Nuo | Wen, Wujie | Venkitasubramaniam, Parv | Ding, Caiwen2023 IEEE Symposium on Security and Privacy (SP), (2023), P.1944
https://doi.org/10.1109/SP46215.2023.10179457 [Citations: 1] -
FSGformer: Frequency Separation and Guidance Transformer for Pansharpening
Liu, Qian | Zhao, Xiangyu | Qin, You | Li, Lanyu | Liu, JunminIEEE Transactions on Geoscience and Remote Sensing, Vol. 63 (2025), Iss. P.1
https://doi.org/10.1109/TGRS.2025.3544755 [Citations: 0] -
No Time to Train: Empowering Non-Parametric Networks for Few-Shot 3D Scene Segmentation
Zhu, Xiangyang | Zhang, Renrui | He, Bowei | Guo, Ziyu | Liu, Jiaming | Xiao, Han | Fu, Chaoyou | Dong, Hao | Gao, Peng2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), (2024), P.3838
https://doi.org/10.1109/CVPR52733.2024.00368 [Citations: 2] -
Bearing Fault Diagnosis Method Based on Attention Mechanism and Multi-Channel Feature Fusion
Gao, Hongfeng | Ma, Jie | Zhang, Zhonghang | Cai, ChaozhiIEEE Access, Vol. 12 (2024), Iss. P.45011
https://doi.org/10.1109/ACCESS.2024.3381618 [Citations: 6] -
Implicit Regularization of Dropout
Zhang, Zhongwang | Xu, Zhi-Qin JohnIEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 46 (2024), Iss. 6 P.4206
https://doi.org/10.1109/TPAMI.2024.3357172 [Citations: 3] -
Overview Frequency Principle/Spectral Bias in Deep Learning
Xu, Zhi-Qin John | Zhang, Yaoyu | Luo, TaoCommunications on Applied Mathematics and Computation, Vol. (2024), Iss.
https://doi.org/10.1007/s42967-024-00398-7 [Citations: 5] -
Probabilistic quantile multiple fourier feature network for lake temperature forecasting: incorporating pinball loss for uncertainty estimation
Liu, Siyuan | Deng, Jiaxin | Yuan, Jin | Li, Weide | Li, Xi’an | Xu, Jing | Zhang, Shaotong | Wu, Jinran | Wang, You-GanEarth Science Informatics, Vol. 17 (2024), Iss. 6 P.5135
https://doi.org/10.1007/s12145-024-01448-7 [Citations: 0] -
Data-driven parametric soliton-rogon state transitions for nonlinear wave equations using deep learning with Fourier neural operator
Zhong, Ming | Yan, Zhenya | Tian, Shou-FuCommunications in Theoretical Physics, Vol. 75 (2023), Iss. 2 P.025001
https://doi.org/10.1088/1572-9494/acab55 [Citations: 5] -
A novel fast spectral computed tomography reconstruction method with spectral windowed analytical iterative technique
Pan, Huiying | Jiang, Xu | Shi, Ligen | Zhang, Wei | Zhao, XingOptics and Lasers in Engineering, Vol. 189 (2025), Iss. P.108935
https://doi.org/10.1016/j.optlaseng.2025.108935 [Citations: 0] -
On understanding and overcoming spectral biases of deep neural network learning methods for solving PDEs
Xu, Zhi-Qin John | Zhang, Lulu | Cai, WeiJournal of Computational Physics, Vol. 530 (2025), Iss. P.113905
https://doi.org/10.1016/j.jcp.2025.113905 [Citations: 0] -
Unsupervised Deep Learning for Ground Roll and Scattered Noise Attenuation
Liu, Dawei | Sacchi, Mauricio D. | Wang, Xiaokai | Chen, WenchaoIEEE Transactions on Geoscience and Remote Sensing, Vol. 61 (2023), Iss. P.1
https://doi.org/10.1109/TGRS.2023.3325324 [Citations: 15] -
Gaussian low‐pass channel attention convolution network for RF fingerprinting
Zhang, Shunjie | Wu, Tianhao | Wang, Wei | Zhan, Ronghui | Zhang, JunElectronics Letters, Vol. 59 (2023), Iss. 12
https://doi.org/10.1049/ell2.12846 [Citations: 1] -
Minimalism is King! High-Frequency Energy-Based Screening for Data-Efficient Backdoor Attacks
Xun, Yuan | Jia, Xiaojun | Gu, Jindong | Liu, Xinwei | Guo, Qing | Cao, XiaochunIEEE Transactions on Information Forensics and Security, Vol. 19 (2024), Iss. P.4560
https://doi.org/10.1109/TIFS.2024.3380821 [Citations: 1] -
Input gradient annealing neural network for solving Fokker-Planck equations with low temperature
Hang, Liangkai | Hu, Dan | Xu, Zhi-Qin JohnJournal of Computational Physics, Vol. 524 (2025), Iss. P.113688
https://doi.org/10.1016/j.jcp.2024.113688 [Citations: 0] -
HMDM: A hybrid GM model-based and MHSA-GRU data-driven method for remaining useful life prediction of rolling bearings
Zhao, Qiwu | Zhang, XiaoliJournal of Vibration and Control, Vol. (2024), Iss.
https://doi.org/10.1177/10775463241273080 [Citations: 0] -
Regularize implicit neural representation by itself
Li, Zhemin | Wang, Hongxia | Meng, Deyu2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), (2023), P.10280
https://doi.org/10.1109/CVPR52729.2023.00991 [Citations: 4] -
Learning Gabor Texture Features for Fine-Grained Recognition
Zhu, Lanyun | Chen, Tianrun | Yin, Jianxiong | See, Simon | Liu, Jun2023 IEEE/CVF International Conference on Computer Vision (ICCV), (2023), P.1621
https://doi.org/10.1109/ICCV51070.2023.00156 [Citations: 13] -
Optimization of Physics-Informed Neural Networks for Solving the Nolinear Schrödinger Equation
Chuprov, I. | Gao, Jiexing | Efremenko, D. | Kazakov, E. | Buzaev, F. | Zemlyakov, V.Doklady Mathematics, Vol. 108 (2023), Iss. S2 P.S186
https://doi.org/10.1134/S1064562423701120 [Citations: 3] -
Towards Inadequately Pre-trained Models in Transfer Learning
Deng, Andong | Li, Xingjian | Hu, Di | Wang, Tianyang | Xiong, Haoyi | Xu, Cheng-Zhong2023 IEEE/CVF International Conference on Computer Vision (ICCV), (2023), P.19340
https://doi.org/10.1109/ICCV51070.2023.01777 [Citations: 4]