arrow
Volume 28, Issue 5
Butterfly-Net: Optimal Function Representation Based on Convolutional Neural Networks

Yingzhou Li, Xiuyuan Cheng & Jianfeng Lu

Commun. Comput. Phys., 28 (2020), pp. 1838-1885.

Published online: 2020-11

Export citation
  • Abstract

Deep networks, especially convolutional neural networks (CNNs), have been successfully applied in various areas of machine learning as well as to challenging problems in other scientific and engineering fields. This paper introduces Butterfly-net, a low-complexity CNN with structured and sparse cross-channel connections, together with a Butterfly initialization strategy for a family of networks. Theoretical analysis of the approximation power of Butterfly-net to the Fourier representation of input data shows that the error decays exponentially as the depth increases. Combining Butterfly-net with a fully connected neural network, a large class of problems are proved to be well approximated with network complexity depending on the effective frequency bandwidth instead of the input dimension. Regular CNN is covered as a special case in our analysis. Numerical experiments validate the analytical results on the approximation of Fourier kernels and energy functionals of Poisson's equations. Moreover, all experiments support that training from Butterfly initialization outperforms training from random initialization. Also, adding the remaining cross-channel connections, although significantly increases the parameter number, does not much improve the post-training accuracy and is more sensitive to data distribution.

  • AMS Subject Headings

15A23, 65D05, 65F10, 62G08, 68W20, 68W25

  • Copyright

COPYRIGHT: © Global Science Press

  • Email address
  • BibTex
  • RIS
  • TXT
@Article{CiCP-28-1838, author = {Li , YingzhouCheng , Xiuyuan and Lu , Jianfeng}, title = {Butterfly-Net: Optimal Function Representation Based on Convolutional Neural Networks}, journal = {Communications in Computational Physics}, year = {2020}, volume = {28}, number = {5}, pages = {1838--1885}, abstract = {

Deep networks, especially convolutional neural networks (CNNs), have been successfully applied in various areas of machine learning as well as to challenging problems in other scientific and engineering fields. This paper introduces Butterfly-net, a low-complexity CNN with structured and sparse cross-channel connections, together with a Butterfly initialization strategy for a family of networks. Theoretical analysis of the approximation power of Butterfly-net to the Fourier representation of input data shows that the error decays exponentially as the depth increases. Combining Butterfly-net with a fully connected neural network, a large class of problems are proved to be well approximated with network complexity depending on the effective frequency bandwidth instead of the input dimension. Regular CNN is covered as a special case in our analysis. Numerical experiments validate the analytical results on the approximation of Fourier kernels and energy functionals of Poisson's equations. Moreover, all experiments support that training from Butterfly initialization outperforms training from random initialization. Also, adding the remaining cross-channel connections, although significantly increases the parameter number, does not much improve the post-training accuracy and is more sensitive to data distribution.

}, issn = {1991-7120}, doi = {https://doi.org/10.4208/cicp.OA-2020-0214}, url = {http://global-sci.org/intro/article_detail/cicp/18398.html} }
TY - JOUR T1 - Butterfly-Net: Optimal Function Representation Based on Convolutional Neural Networks AU - Li , Yingzhou AU - Cheng , Xiuyuan AU - Lu , Jianfeng JO - Communications in Computational Physics VL - 5 SP - 1838 EP - 1885 PY - 2020 DA - 2020/11 SN - 28 DO - http://doi.org/10.4208/cicp.OA-2020-0214 UR - https://global-sci.org/intro/article_detail/cicp/18398.html KW - Butterfly algorithm, convolutional neural network, Fourier analysis, deep learning. AB -

Deep networks, especially convolutional neural networks (CNNs), have been successfully applied in various areas of machine learning as well as to challenging problems in other scientific and engineering fields. This paper introduces Butterfly-net, a low-complexity CNN with structured and sparse cross-channel connections, together with a Butterfly initialization strategy for a family of networks. Theoretical analysis of the approximation power of Butterfly-net to the Fourier representation of input data shows that the error decays exponentially as the depth increases. Combining Butterfly-net with a fully connected neural network, a large class of problems are proved to be well approximated with network complexity depending on the effective frequency bandwidth instead of the input dimension. Regular CNN is covered as a special case in our analysis. Numerical experiments validate the analytical results on the approximation of Fourier kernels and energy functionals of Poisson's equations. Moreover, all experiments support that training from Butterfly initialization outperforms training from random initialization. Also, adding the remaining cross-channel connections, although significantly increases the parameter number, does not much improve the post-training accuracy and is more sensitive to data distribution.

Yingzhou Li, Xiuyuan Cheng & Jianfeng Lu. (2020). Butterfly-Net: Optimal Function Representation Based on Convolutional Neural Networks. Communications in Computational Physics. 28 (5). 1838-1885. doi:10.4208/cicp.OA-2020-0214
Copy to clipboard
The citation has been copied to your clipboard