Theory of the Frequency Principle for General Deep Neural Networks (CSIAM Transactions on Applied Mathematics, 2021)
Along with fruitful applications of Deep Neural Networks (DNNs) to realistic problems, recently, empirical studies reported a universal phenomenon of Frequency Principle (F-Principle), that is, a DNN tends to learn a target function fromlow to high frequencies during the training. The F-Principle has been very useful inproviding both qualitative and quantitative understandings of DNNs. In this paper,we rigorously investigate the F-Principle for the training dynamics of a general DNNat three stages: initial stage, intermediate stage, and final stage. For each stage, a theorem is provided in terms of proper quantities characterizing the F-Principle. Ourresults are general in the sense that they work for multilayer networks with generalactivation functions, population densities of data, and a large class of loss functions.Our work lays a theoretical foundation of the F-Principle for a better understanding ofthe training process of DNNs.
Keywords: Frequency principle; Deep Neural Networks; dynamical system; training process.