On the Banach Spaces Associated with Multi-Layer ReLU Networks: Function Representation, Approximation Theory and Gradient Descent Dynamics
Year: 2020
Author: Weinan E, Stephan Wojtowytsch
CSIAM Transactions on Applied Mathematics, Vol. 1 (2020), Iss. 3 : pp. 387–440
Abstract
We develop Banach spaces for ReLU neural networks of finite depth $L$ and
infinite width. The spaces contain all finite fully connected $L$-layer networks and their $L^2$-limiting objects under bounds on the natural path-norm. Under this norm, the
unit ball in the space for $L$-layer networks has low Rademacher complexity and thus
favorable generalization properties. Functions in these spaces can be approximated by
multi-layer neural networks with dimension-independent convergence rates.
The key to this work is a new way of representing functions in some form of expectations, motivated by multi-layer neural networks. This representation allows us to
define a new class of continuous models for machine learning. We show that the gradient flow defined this way is the natural continuous analog of the gradient descent
dynamics for the associated multi-layer neural networks. We show that the path-norm
increases at most polynomially under this continuous gradient flow dynamics.
You do not have full access to this article.
Already a Subscriber? Sign in as an individual or via your institution
Journal Article Details
Publisher Name: Global Science Press
Language: English
DOI: https://doi.org/10.4208/csiam-am.20-211
CSIAM Transactions on Applied Mathematics, Vol. 1 (2020), Iss. 3 : pp. 387–440
Published online: 2020-01
AMS Subject Headings: Global Science Press
Copyright: COPYRIGHT: © Global Science Press
Pages: 54
Keywords: Barron space multi-layer space deep neural network representations of functions machine learning infinitely wide network ReLU activation Banach space path-norm continuous gradient descent dynamics index representation.