Close Search Advanced
Journals
Resources
About Us
Open Access

Deep Network Approximation Characterized by Number of Neurons

Deep Network Approximation Characterized by Number of Neurons
Preview
Add to basket

Year:    2020

Author:    Zuowei Shen, Haizhao Yang, Shijun Zhang

Communications in Computational Physics, Vol. 28 (2020), Iss. 5 : pp. 1768–1811

Abstract

This paper quantitatively characterizes the approximation power of deep feed-forward neural networks (FNNs) in terms of the number of neurons. It is shown by construction that ReLU FNNs with width $\mathcal{O}$(max{$d⌊N^{1/d}⌋$,$N$+1}) and depth $\mathcal{O}(L)$ can approximate an arbitrary Hölder continuous function of order $α∈(0,1]$ on $[0,1]^d$ with a nearly tight approximation rate $\mathcal{O}(\sqrt{d}N^{−2α/d}L^{−2α/d})$ measured in $L^p$ -norm for any $N,L∈\mathbb{N}^+$ and $p∈[1,∞]$. More generally for an arbitrary continuous function $f$ on $[0,1]^d$ with a modulus of continuity $ω_f (·)$, the constructive approximation rate is $\mathcal{O}(\sqrt{d}ω_f(N^{−2α/d}L^{−2α/d}))$. We also extend our analysis to $f$ on irregular domains or those localized in an ε-neighborhood of a $d_\mathcal{M}$-dimensional smooth manifold $\mathcal{M}⊆[0,1]^d$ with $d_\mathcal{M}≪d$. Especially, in the case of an essentially low-dimensional domain, we show an approximation rate $\mathcal{O}(ω_f(\frac{ε}{1−δ}\sqrt{\frac{d}{d_δ}}+ε)+\sqrt{d}ω_f(\frac{\sqrt{d}}{1−δ\sqrt{d_δ}}N^{−2α/d_δ}L^{−2α/d_δ})$ for ReLU FNNs to approximate $f$ in the $ε$-neighborhood, where $d_δ=\mathcal{O}(d_\mathcal{M}\frac{\rm{ln}(d/δ)}{δ^2})$ for any $δ∈(0,1)$ as a relative error for a projection to approximate an isometry when projecting $\mathcal{M}$ to a $d_δ$-dimensional domain.

Submit Article

You do not have full access to this article.

Already a Subscriber? Sign in as an individual or via your institution

Journal Article Details

Publisher Name:    Global Science Press

Language:    English

DOI:    https://doi.org/10.4208/cicp.OA-2020-0149

Communications in Computational Physics, Vol. 28 (2020), Iss. 5 : pp. 1768–1811

Published online:    2020-01

AMS Subject Headings:    Global Science Press

Copyright:    COPYRIGHT: © Global Science Press

Pages:    44

Keywords:    Deep ReLU neural networks Hölder continuity modulus of continuity approximation theory low-dimensional manifold parallel computing.

Author Details

Zuowei Shen

Haizhao Yang

Shijun Zhang

  1. Uniform Convergence of Deep Neural Networks With Lipschitz Continuous Activation Functions and Variable Widths

    Xu, Yuesheng | Zhang, Haizhang

    IEEE Transactions on Information Theory, Vol. 70 (2024), Iss. 10 P.7125

    https://doi.org/10.1109/TIT.2024.3439136 [Citations: 0]

  2. Deep Dimension Reduction for Supervised Representation Learning

    Huang, Jian | Jiao, Yuling | Liao, Xu | Liu, Jin | Yu, Zhou

    IEEE Transactions on Information Theory, Vol. 70 (2024), Iss. 5 P.3583

    https://doi.org/10.1109/TIT.2023.3340658 [Citations: 3]

  3. An operator learning perspective on parameter-to-observable maps

    Huang, Daniel Zhengyu | Nelsen, Nicholas H. | Trautner, Margaret

    Foundations of Data Science, Vol. 0 (2024), Iss. 0 P.0

    https://doi.org/10.3934/fods.2024037 [Citations: 0]

  4. Deep Network Approximation for Smooth Functions

    Lu, Jianfeng | Shen, Zuowei | Yang, Haizhao | Zhang, Shijun

    SIAM Journal on Mathematical Analysis, Vol. 53 (2021), Iss. 5 P.5465

    https://doi.org/10.1137/20M134695X [Citations: 64]

  5. Least-Squares Neural Network (LSNN) Method for Linear Advection-Reaction Equation: Discontinuity Interface

    Cai, Zhiqiang | Choi, Junpyo | Liu, Min

    SIAM Journal on Scientific Computing, Vol. 46 (2024), Iss. 4 P.C448

    https://doi.org/10.1137/23M1568107 [Citations: 0]

  6. Optimal Rates of Approximation by Shallow ReLU$$^k$$ Neural Networks and Applications to Nonparametric Regression

    Yang, Yunfei | Zhou, Ding-Xuan

    Constructive Approximation, Vol. (2024), Iss.

    https://doi.org/10.1007/s00365-024-09679-z [Citations: 0]

  7. How do noise tails impact on deep ReLU networks?

    Fan, Jianqing | Gu, Yihong | Zhou, Wen-Xin

    The Annals of Statistics, Vol. 52 (2024), Iss. 4

    https://doi.org/10.1214/24-AOS2428 [Citations: 0]

  8. Convergence of deep convolutional neural networks

    Xu, Yuesheng | Zhang, Haizhang

    Neural Networks, Vol. 153 (2022), Iss. P.553

    https://doi.org/10.1016/j.neunet.2022.06.031 [Citations: 23]

  9. The Kolmogorov–Arnold representation theorem revisited

    Schmidt-Hieber, Johannes

    Neural Networks, Vol. 137 (2021), Iss. P.119

    https://doi.org/10.1016/j.neunet.2021.01.020 [Citations: 36]

  10. Spline representation and redundancies of one-dimensional ReLU neural network models

    Plonka, Gerlind | Riebe, Yannick | Kolomoitsev, Yurii

    Analysis and Applications, Vol. 21 (2023), Iss. 01 P.127

    https://doi.org/10.1142/S0219530522400103 [Citations: 3]

  11. Convergence analysis of deep residual networks

    Huang, Wentao | Zhang, Haizhang

    Analysis and Applications, Vol. 22 (2024), Iss. 02 P.351

    https://doi.org/10.1142/S021953052350029X [Citations: 1]

  12. Auto- Train-Once: Controller Network Guided Automatic Network Pruning from Scratch

    Wu, Xidong | Gao, Shangqian | Zhang, Zeyu | Li, Zhenzhen | Bao, Runxue | Zhang, Yanfu | Wang, Xiaoqian | Huang, Heng

    2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), (2024), P.16163

    https://doi.org/10.1109/CVPR52733.2024.01530 [Citations: 0]

  13. Discovery of Subdiffusion Problem with Noisy Data via Deep Learning

    Xu, Xingjian | Chen, Minghua

    Journal of Scientific Computing, Vol. 92 (2022), Iss. 1

    https://doi.org/10.1007/s10915-022-01879-8 [Citations: 3]

  14. Optimal approximation rate of ReLU networks in terms of width and depth

    Shen, Zuowei | Yang, Haizhao | Zhang, Shijun

    Journal de Mathématiques Pures et Appliquées, Vol. 157 (2022), Iss. P.101

    https://doi.org/10.1016/j.matpur.2021.07.009 [Citations: 31]

  15. Full error analysis for the training of deep neural networks

    Beck, Christian | Jentzen, Arnulf | Kuckuck, Benno

    Infinite Dimensional Analysis, Quantum Probability and Related Topics, Vol. 25 (2022), Iss. 02

    https://doi.org/10.1142/S021902572150020X [Citations: 12]

  16. A Deep Generative Approach to Conditional Sampling

    Zhou, Xingyu | Jiao, Yuling | Liu, Jin | Huang, Jian

    Journal of the American Statistical Association, Vol. 118 (2023), Iss. 543 P.1837

    https://doi.org/10.1080/01621459.2021.2016424 [Citations: 7]

  17. Deep nonlinear sufficient dimension reduction

    Chen, YinFeng | Jiao, YuLing | Qiu, Rui | Yu, Zhou

    The Annals of Statistics, Vol. 52 (2024), Iss. 3

    https://doi.org/10.1214/24-AOS2390 [Citations: 0]

  18. Deep Neural Networks for Solving Large Linear Systems Arising from High-Dimensional Problems

    Gu, Yiqi | Ng, Michael K.

    SIAM Journal on Scientific Computing, Vol. 45 (2023), Iss. 5 P.A2356

    https://doi.org/10.1137/22M1488132 [Citations: 5]

  19. A comparative investigation of neural networks in solving differential equations

    Shi, Enze | Xu, Chuanju

    Journal of Algorithms & Computational Technology, Vol. 15 (2021), Iss.

    https://doi.org/10.1177/1748302621998605 [Citations: 10]

  20. Low dimensional approximation and generalization of multivariate functions on smooth manifolds using deep ReLU neural networks

    Labate, Demetrio | Shi, Ji

    Neural Networks, Vol. 174 (2024), Iss. P.106223

    https://doi.org/10.1016/j.neunet.2024.106223 [Citations: 1]

  21. Simultaneous neural network approximation for smooth functions

    Hon, Sean | Yang, Haizhao

    Neural Networks, Vol. 154 (2022), Iss. P.152

    https://doi.org/10.1016/j.neunet.2022.06.040 [Citations: 7]

  22. Deep estimation for Q⁎ with minimax Bellman error minimization

    Kang, Lican | Liao, Xu | Liu, Jin | Luo, Yuan

    Information Sciences, Vol. 648 (2023), Iss. P.119565

    https://doi.org/10.1016/j.ins.2023.119565 [Citations: 0]

  23. Classification Logit Two-Sample Testing by Neural Networks for Differentiating Near Manifold Densities

    Cheng, Xiuyuan | Cloninger, Alexander

    IEEE Transactions on Information Theory, Vol. 68 (2022), Iss. 10 P.6631

    https://doi.org/10.1109/TIT.2022.3175691 [Citations: 3]

  24. Approximation bounds for norm constrained neural networks with applications to regression and GANs

    Jiao, Yuling | Wang, Yang | Yang, Yunfei

    Applied and Computational Harmonic Analysis, Vol. 65 (2023), Iss. P.249

    https://doi.org/10.1016/j.acha.2023.03.004 [Citations: 3]

  25. Solving PDEs on unknown manifolds with machine learning

    Liang, Senwei | Jiang, Shixiao W. | Harlim, John | Yang, Haizhao

    Applied and Computational Harmonic Analysis, Vol. 71 (2024), Iss. P.101652

    https://doi.org/10.1016/j.acha.2024.101652 [Citations: 0]

  26. Коллокационная аппроксимация глубокими $\mathrm{ReLU}$-нейронными сетями решений параметрических и стохастических уравнений с частными производными c логнормальными входами

    Dinh, Dung

    Математический сборник, Vol. 214 (2023), Iss. 4 P.38

    https://doi.org/10.4213/sm9791 [Citations: 0]

  27. SignReLU neural network and its approximation ability

    Li, Jianfei | Feng, Han | Zhou, Ding-Xuan

    Journal of Computational and Applied Mathematics, Vol. 440 (2024), Iss. P.115551

    https://doi.org/10.1016/j.cam.2023.115551 [Citations: 1]
  28. Numerical Analysis Meets Machine Learning

    Two-layer neural networks for partial differential equations: optimization and generalization theory

    Luo, Tao | Yang, Haizhao

    2024

    https://doi.org/10.1016/bs.hna.2024.05.007 [Citations: 0]

  29. Neural network approximation: Three hidden layers are enough

    Shen, Zuowei | Yang, Haizhao | Zhang, Shijun

    Neural Networks, Vol. 141 (2021), Iss. P.160

    https://doi.org/10.1016/j.neunet.2021.04.011 [Citations: 55]

  30. Deep Network With Approximation Error Being Reciprocal of Width to Power of Square Root of Depth

    Shen, Zuowei | Yang, Haizhao | Zhang, Shijun

    Neural Computation, Vol. 33 (2021), Iss. 4 P.1005

    https://doi.org/10.1162/neco_a_01364 [Citations: 25]

  31. Approximation analysis of CNNs from a feature extraction view

    Li, Jianfei | Feng, Han | Zhou, Ding-Xuan

    Analysis and Applications, Vol. 22 (2024), Iss. 03 P.635

    https://doi.org/10.1142/S0219530524500015 [Citations: 1]

  32. Novel GA-Based DNN Architecture for Identifying the Failure Mode with High Accuracy and Analyzing Its Effects on the System

    Rezaeian, Naeim | Gurina, Regina | Saltykova, Olga A. | Hezla, Lokmane | Nohurov, Mammetnazar | Reza Kashyzadeh, Kazem

    Applied Sciences, Vol. 14 (2024), Iss. 8 P.3354

    https://doi.org/10.3390/app14083354 [Citations: 8]

  33. Deep regression learning with optimal loss function

    Wang, Xuancheng | Zhou, Ling | Lin, Huazhen

    Journal of the American Statistical Association, Vol. (2024), Iss. P.1

    https://doi.org/10.1080/01621459.2024.2412364 [Citations: 0]

  34. Deep Neural Networks with ReLU-Sine-Exponential Activations Break Curse of Dimensionality in Approximation on Hölder Class

    Jiao, Yuling | Lai, Yanming | Lu, Xiliang | Wang, Fengru | Yang, Jerry Zhijian | Yang, Yuanyuan

    SIAM Journal on Mathematical Analysis, Vol. 55 (2023), Iss. 4 P.3635

    https://doi.org/10.1137/21M144431X [Citations: 1]

  35. The Discovery of Dynamics via Linear Multistep Methods and Deep Learning: Error Estimation

    Du, Qiang | Gu, Yiqi | Yang, Haizhao | Zhou, Chao

    SIAM Journal on Numerical Analysis, Vol. 60 (2022), Iss. 4 P.2014

    https://doi.org/10.1137/21M140691X [Citations: 11]

  36. Simultaneous approximation of a smooth function and its derivatives by deep neural networks with piecewise-polynomial activations

    Belomestny, Denis | Naumov, Alexey | Puchkin, Nikita | Samsonov, Sergey

    Neural Networks, Vol. 161 (2023), Iss. P.242

    https://doi.org/10.1016/j.neunet.2023.01.035 [Citations: 7]

  37. The Gap between Theory and Practice in Function Approximation with Deep Neural Networks

    Adcock, Ben | Dexter, Nick

    SIAM Journal on Mathematics of Data Science, Vol. 3 (2021), Iss. 2 P.624

    https://doi.org/10.1137/20M131309X [Citations: 40]

  38. Stochastic Markov gradient descent and training low-bit neural networks

    Ashbrock, Jonathan | Powell, Alexander M.

    Sampling Theory, Signal Processing, and Data Analysis, Vol. 19 (2021), Iss. 2

    https://doi.org/10.1007/s43670-021-00015-1 [Citations: 1]

  39. Collocation approximation by deep neural ReLU networks for parametric and stochastic PDEs with lognormal inputs

    Dinh, Dung

    Sbornik: Mathematics, Vol. 214 (2023), Iss. 4 P.479

    https://doi.org/10.4213/sm9791e [Citations: 2]

  40. Active learning based sampling for high-dimensional nonlinear partial differential equations

    Gao, Wenhan | Wang, Chunmei

    Journal of Computational Physics, Vol. 475 (2023), Iss. P.111848

    https://doi.org/10.1016/j.jcp.2022.111848 [Citations: 12]

  41. On the capacity of deep generative networks for approximating distributions

    Yang, Yunfei | Li, Zhen | Wang, Yang

    Neural Networks, Vol. 145 (2022), Iss. P.144

    https://doi.org/10.1016/j.neunet.2021.10.012 [Citations: 5]

  42. Understanding the Performance of Learning Precoding Policies With Graph and Convolutional Neural Networks

    Zhao, Baichuan | Guo, Jia | Yang, Chenyang

    IEEE Transactions on Communications, Vol. 72 (2024), Iss. 9 P.5657

    https://doi.org/10.1109/TCOMM.2024.3388506 [Citations: 0]

  43. Deep nonparametric regression on approximate manifolds: Nonasymptotic error bounds with polynomial prefactors

    Jiao, Yuling | Shen, Guohao | Lin, Yuanyuan | Huang, Jian

    The Annals of Statistics, Vol. 51 (2023), Iss. 2

    https://doi.org/10.1214/23-AOS2266 [Citations: 10]

  44. NeuroSketch: Fast and Approximate Evaluation of Range Aggregate Queries with Neural Networks

    Zeighami, Sepanta | Shahabi, Cyrus | Sharan, Vatsal

    Proceedings of the ACM on Management of Data, Vol. 1 (2023), Iss. 1 P.1

    https://doi.org/10.1145/3588954 [Citations: 6]

  45. Convergence of deep ReLU networks

    Xu, Yuesheng | Zhang, Haizhang

    Neurocomputing, Vol. 571 (2024), Iss. P.127174

    https://doi.org/10.1016/j.neucom.2023.127174 [Citations: 7]

  46. Approximation of smooth functionals using deep ReLU networks

    Song, Linhao | Liu, Ying | Fan, Jun | Zhou, Ding-Xuan

    Neural Networks, Vol. 166 (2023), Iss. P.424

    https://doi.org/10.1016/j.neunet.2023.07.012 [Citations: 7]

  47. An approximation theory framework for measure-transport sampling algorithms

    Baptista, Ricardo | Hosseini, Bamdad | Kovachki, Nikola | Marzouk, Youssef | Sagiv, Amir

    Mathematics of Computation, Vol. (2024), Iss.

    https://doi.org/10.1090/mcom/4013 [Citations: 1]

  48. Strong Overall Error Analysis for the Training of Artificial Neural Networks Via Random Initializations

    Jentzen, Arnulf | Riekert, Adrian

    Communications in Mathematics and Statistics, Vol. 12 (2024), Iss. 3 P.385

    https://doi.org/10.1007/s40304-022-00292-9 [Citations: 0]

  49. Approximation in shift-invariant spaces with deep ReLU neural networks

    Yang, Yunfei | Li, Zhen | Wang, Yang

    Neural Networks, Vol. 153 (2022), Iss. P.269

    https://doi.org/10.1016/j.neunet.2022.06.013 [Citations: 3]

  50. Computing ground states of Bose-Einstein condensation by normalized deep neural network

    Bao, Weizhu | Chang, Zhipeng | Zhao, Xiaofei

    Journal of Computational Physics, Vol. 520 (2025), Iss. P.113486

    https://doi.org/10.1016/j.jcp.2024.113486 [Citations: 0]

  51. Deep ReLU networks and high-order finite element methods II: Chebyšev emulation

    Opschoor, Joost A.A. | Schwab, Christoph

    Computers & Mathematics with Applications, Vol. 169 (2024), Iss. P.142

    https://doi.org/10.1016/j.camwa.2024.06.008 [Citations: 0]

  52. Deep Learning Solution of the Eigenvalue Problem for Differential Operators

    Ben-Shaul, Ido | Bar, Leah | Fishelov, Dalia | Sochen, Nir

    Neural Computation, Vol. 35 (2023), Iss. 6 P.1100

    https://doi.org/10.1162/neco_a_01583 [Citations: 6]

  53. PolyARBerNN: A Neural Network Guided Solver and Optimizer for Bounded Polynomial Inequalities

    Fatnassi, Wael | Shoukry, Yasser

    ACM Transactions on Embedded Computing Systems, Vol. 23 (2024), Iss. 2 P.1

    https://doi.org/10.1145/3632970 [Citations: 0]