TY - JOUR
T1 - Theory of the Frequency Principle for General Deep Neural Networks
AU - Luo , Tao
AU - Ma , Zheng
AU - Xu , Zhi-Qin John
AU - Zhang , Yaoyu
JO - CSIAM Transactions on Applied Mathematics
VL - 3
SP - 484
EP - 507
PY - 2021
DA - 2021/08
SN - 2
DO - http://doi.org/10.4208/csiam-am.SO-2020-0005
UR - https://global-sci.org/intro/article_detail/csiam-am/19447.html
KW - Frequency principle, Deep Neural Networks, dynamical system, training process.
AB - <p style="text-align: justify;">Along with fruitful applications of Deep Neural Networks (DNNs) to realistic problems, recently, empirical studies reported a universal phenomenon of Frequency Principle (F-Principle), that is, a DNN tends to learn a target function from
low to high frequencies during the training. The F-Principle has been very useful in
providing both qualitative and quantitative understandings of DNNs. In this paper,
we rigorously investigate the F-Principle for the training dynamics of a general DNN
at three stages: initial stage, intermediate stage, and final stage. For each stage, a theorem is provided in terms of proper quantities characterizing the F-Principle. Our
results are general in the sense that they work for multilayer networks with general
activation functions, population densities of data, and a large class of loss functions.
Our work lays a theoretical foundation of the F-Principle for a better understanding of
the training process of DNNs.</p>