Close Search Advanced
Journals
Resources
About Us
Open Access

Better Approximations of High Dimensional Smooth Functions by Deep Neural Networks with Rectified Power Units

Better Approximations of High Dimensional Smooth Functions by Deep Neural Networks with Rectified Power Units
Preview
Add to basket

Year:    2020

Author:    Bo Li, Shanshan Tang, Haijun Yu

Communications in Computational Physics, Vol. 27 (2020), Iss. 2 : pp. 379–411

Abstract

Deep neural networks with rectified linear units (ReLU) are getting more and more popular due to their universal representation power and successful applications. Some theoretical progress regarding the approximation power of deep ReLU network for functions in Sobolev space and Korobov space have recently been made by [D. Yarotsky, Neural Network, 94:103-114, 2017] and [H. Montanelli and Q. Du, SIAM J Math. Data Sci., 1:78-92, 2019], etc. In this paper, we show that deep networks with rectified power units (RePU) can give better approximations for smooth functions than deep ReLU networks. Our analysis bases on classical polynomial approximation theory and some efficient algorithms proposed in this paper to convert polynomials into deep RePU networks of optimal size with no approximation error. Comparing to the results on ReLU networks, the sizes of RePU networks required to approximate functions in Sobolev space and Korobov space with an error tolerance ε, by our constructive proofs, are in general $O$($log\frac{1}{ε}$) times smaller than the sizes of corresponding ReLU networks constructed in most of the existing literature. Comparing to the classical results of Mhaskar [Mhaskar, Adv. Comput. Math. 1:61-80, 1993], our constructions use less number of activation functions and numerically more stable, they can be served as good initials of deep RePU networks and further trained to break the limit of linear approximation theory. The functions represented by RePU networks are smooth functions, so they naturally fit in the places where derivatives are involved in the loss function.

Submit Article

You do not have full access to this article.

Already a Subscriber? Sign in as an individual or via your institution

Journal Article Details

Publisher Name:    Global Science Press

Language:    English

DOI:    https://doi.org/10.4208/cicp.OA-2019-0168

Communications in Computational Physics, Vol. 27 (2020), Iss. 2 : pp. 379–411

Published online:    2020-01

AMS Subject Headings:    Global Science Press

Copyright:    COPYRIGHT: © Global Science Press

Pages:    33

Keywords:    Deep neural network high dimensional approximation sparse grids rectified linear unit rectified power unit rectified quadratic unit.

Author Details

Bo Li

Shanshan Tang

Haijun Yu

  1. Theoretical guarantees for neural control variates in MCMC

    Belomestny, Denis | Goldman, Artur | Naumov, Alexey | Samsonov, Sergey

    Mathematics and Computers in Simulation, Vol. 220 (2024), Iss. P.382

    https://doi.org/10.1016/j.matcom.2024.01.019 [Citations: 2]

  2. q-Neurons: Neuron Activations Based on Stochastic Jackson’s Derivative Operators

    Nielsen, Frank | Sun, Ke

    IEEE Transactions on Neural Networks and Learning Systems, Vol. 32 (2021), Iss. 6 P.2782

    https://doi.org/10.1109/TNNLS.2020.3005167 [Citations: 1]

  3. Neural Control of Parametric Solutions for High-Dimensional Evolution PDEs

    Gaby, Nathan | Ye, Xiaojing | Zhou, Haomin

    SIAM Journal on Scientific Computing, Vol. 46 (2024), Iss. 2 P.C155

    https://doi.org/10.1137/23M1549870 [Citations: 1]

  4. Simultaneous approximation of a smooth function and its derivatives by deep neural networks with piecewise-polynomial activations

    Belomestny, Denis | Naumov, Alexey | Puchkin, Nikita | Samsonov, Sergey

    Neural Networks, Vol. 161 (2023), Iss. P.242

    https://doi.org/10.1016/j.neunet.2023.01.035 [Citations: 7]

  5. Approximation results for Gradient Flow Trained Shallow Neural Networks in 1d

    Gentile, Russell | Welper, Gerrit

    Constructive Approximation, Vol. 60 (2024), Iss. 3 P.547

    https://doi.org/10.1007/s00365-024-09694-0 [Citations: 0]

  6. Lyapunov-Net: A Deep Neural Network Architecture for Lyapunov Function Approximation

    Gaby, Nathan | Zhang, Fumin | Ye, Xiaojing

    2022 IEEE 61st Conference on Decision and Control (CDC), (2022), P.2091

    https://doi.org/10.1109/CDC51059.2022.9993006 [Citations: 21]

  7. Approximation rates for neural networks with encodable weights in smoothness spaces

    Gühring, Ingo | Raslan, Mones

    Neural Networks, Vol. 134 (2021), Iss. P.107

    https://doi.org/10.1016/j.neunet.2020.11.010 [Citations: 40]

  8. Near-optimal learning of Banach-valued, high-dimensional functions via deep neural networks

    Adcock, Ben | Brugiapaglia, Simone | Dexter, Nick | Moraga, Sebastian

    Neural Networks, Vol. 181 (2025), Iss. P.106761

    https://doi.org/10.1016/j.neunet.2024.106761 [Citations: 0]

  9. The robust physics-informed neural networks for a typical fourth-order phase field model

    Zhang, Wen | Li, Jian

    Computers & Mathematics with Applications, Vol. 140 (2023), Iss. P.64

    https://doi.org/10.1016/j.camwa.2023.03.016 [Citations: 2]

  10. Efficient Spectral Methods for Quasi-Equilibrium Closure Approximations of Symmetric Problems on Unit Circle and Sphere

    Jiang, Shan | Yu, Haijun

    Journal of Scientific Computing, Vol. 89 (2021), Iss. 2

    https://doi.org/10.1007/s10915-021-01646-1 [Citations: 3]

  11. Approximation Algorithm Applications in Artificial Intelligence: A Review

    Abdul Ameer, Hawraa R. | Jaddoa, Ahmed Sami | AL-Challabi, Hawraa A. | Abdul-Zahra, Dalael Saad

    2022 Fifth College of Science International Conference of Recent Trends in Information Technology (CSCTIT), (2022), P.43

    https://doi.org/10.1109/CSCTIT56299.2022.10145686 [Citations: 0]

  12. Solving Parametric Partial Differential Equations with Deep Rectified Quadratic Unit Neural Networks

    Lei, Zhen | Shi, Lei | Zeng, Chenyu

    Journal of Scientific Computing, Vol. 93 (2022), Iss. 3

    https://doi.org/10.1007/s10915-022-02015-2 [Citations: 5]
  13. Uncertainty, Constraints, and Decision Making

    Why Rectified Power (RePU) Activation Functions are Efficient in Deep Learning: A Theoretical Explanation

    Bokati, Laxman | Kreinovich, Vladik | Baca, Joseph | Rovelli, Natasha

    2023

    https://doi.org/10.1007/978-3-031-36394-8_2 [Citations: 0]

  14. Smooth Function Approximation by Deep Neural Networks with General Activation Functions

    Ohn, Ilsang | Kim, Yongdai

    Entropy, Vol. 21 (2019), Iss. 7 P.627

    https://doi.org/10.3390/e21070627 [Citations: 46]

  15. Discovering Phase Field Models from Image Data with the Pseudo-Spectral Physics Informed Neural Networks

    Zhao, Jia

    Communications on Applied Mathematics and Computation, Vol. 3 (2021), Iss. 2 P.357

    https://doi.org/10.1007/s42967-020-00105-2 [Citations: 5]

  16. Constructing custom thermodynamics using deep learning

    Chen, Xiaoli | Soh, Beatrice W. | Ooi, Zi-En | Vissol-Gaudin, Eleonore | Yu, Haijun | Novoselov, Kostya S. | Hippalgaonkar, Kedar | Li, Qianxiao

    Nature Computational Science, Vol. 4 (2023), Iss. 1 P.66

    https://doi.org/10.1038/s43588-023-00581-5 [Citations: 1]

  17. A convergent Deep Learning algorithm for approximation of polynomials

    Després, Bruno

    Comptes Rendus. Mathématique, Vol. 361 (2023), Iss. G6 P.1029

    https://doi.org/10.5802/crmath.462 [Citations: 0]
  18. Numerical Analysis Meets Machine Learning

    Learning smooth functions in high dimensions

    Adcock, Ben | Brugiapaglia, Simone | Dexter, Nick | Moraga, Sebastian

    2024

    https://doi.org/10.1016/bs.hna.2024.05.001 [Citations: 1]

  19. Sparse Approximation of Triangular Transports, Part I: The Finite-Dimensional Case

    Zech, Jakob | Marzouk, Youssef

    Constructive Approximation, Vol. 55 (2022), Iss. 3 P.919

    https://doi.org/10.1007/s00365-022-09569-2 [Citations: 5]

  20. OnsagerNet: Learning stable and interpretable dynamics using a generalized Onsager principle

    Yu, Haijun | Tian, Xinyuan | E, Weinan | Li, Qianxiao

    Physical Review Fluids, Vol. 6 (2021), Iss. 11

    https://doi.org/10.1103/PhysRevFluids.6.114402 [Citations: 25]

  21. Constructive Deep ReLU Neural Network Approximation

    Herrmann, Lukas | Opschoor, Joost A. A. | Schwab, Christoph

    Journal of Scientific Computing, Vol. 90 (2022), Iss. 2

    https://doi.org/10.1007/s10915-021-01718-2 [Citations: 12]

  22. Optimal approximation of infinite-dimensional holomorphic functions

    Adcock, Ben | Dexter, Nick | Moraga, Sebastian

    Calcolo, Vol. 61 (2024), Iss. 1

    https://doi.org/10.1007/s10092-023-00565-x [Citations: 3]

  23. ChebNet: Efficient and Stable Constructions of Deep Neural Networks with Rectified Power Units via Chebyshev Approximation

    Tang, Shanshan | Li, Bo | Yu, Haijun

    Communications in Mathematics and Statistics, Vol. (2024), Iss.

    https://doi.org/10.1007/s40304-023-00392-0 [Citations: 0]

  24. Exponential ReLU DNN Expression of Holomorphic Maps in High Dimension

    Opschoor, J. A. A. | Schwab, Ch. | Zech, J.

    Constructive Approximation, Vol. 55 (2022), Iss. 1 P.537

    https://doi.org/10.1007/s00365-021-09542-5 [Citations: 33]

  25. Two-layer networks with the $$\text {ReLU}^k$$ activation function: Barron spaces and derivative approximation

    Li, Yuanyuan | Lu, Shuai | Mathé, Peter | Pereverzev, Sergei V.

    Numerische Mathematik, Vol. 156 (2024), Iss. 1 P.319

    https://doi.org/10.1007/s00211-023-01384-6 [Citations: 1]