Year: 2020
Author: Yingzhou Li, Xiuyuan Cheng, Jianfeng Lu
Communications in Computational Physics, Vol. 28 (2020), Iss. 5 : pp. 1838–1885
Abstract
Deep networks, especially convolutional neural networks (CNNs), have been successfully applied in various areas of machine learning as well as to challenging problems in other scientific and engineering fields. This paper introduces Butterfly-net, a low-complexity CNN with structured and sparse cross-channel connections, together with a Butterfly initialization strategy for a family of networks. Theoretical analysis of the approximation power of Butterfly-net to the Fourier representation of input data shows that the error decays exponentially as the depth increases. Combining Butterfly-net with a fully connected neural network, a large class of problems are proved to be well approximated with network complexity depending on the effective frequency bandwidth instead of the input dimension. Regular CNN is covered as a special case in our analysis. Numerical experiments validate the analytical results on the approximation of Fourier kernels and energy functionals of Poisson's equations. Moreover, all experiments support that training from Butterfly initialization outperforms training from random initialization. Also, adding the remaining cross-channel connections, although significantly increases the parameter number, does not much improve the post-training accuracy and is more sensitive to data distribution.
You do not have full access to this article.
Already a Subscriber? Sign in as an individual or via your institution
Journal Article Details
Publisher Name: Global Science Press
Language: English
DOI: https://doi.org/10.4208/cicp.OA-2020-0214
Communications in Computational Physics, Vol. 28 (2020), Iss. 5 : pp. 1838–1885
Published online: 2020-01
AMS Subject Headings: Global Science Press
Copyright: COPYRIGHT: © Global Science Press
Pages: 48
Keywords: Butterfly algorithm convolutional neural network Fourier analysis deep learning.
Author Details
-
Structured Matrices and Their Application in Neural Networks: A Survey
Kissel, Matthias | Diepold, KlausNew Generation Computing, Vol. 41 (2023), Iss. 3 P.697
https://doi.org/10.1007/s00354-023-00226-1 [Citations: 2] -
Growth Factors of Random Butterfly Matrices and the Stability of Avoiding Pivoting
Peca-Medlin, John | Trogdon, ThomasSIAM Journal on Matrix Analysis and Applications, Vol. 44 (2023), Iss. 3 P.945
https://doi.org/10.1137/22M148762X [Citations: 2]