Year: 2024
Author: Steffen Dereich, Sebastian Kassing
Journal of Machine Learning, Vol. 3 (2024), Iss. 1 : pp. 1–22
Abstract
We prove existence of global minima in the loss landscape for the approximation of continuous target functions using shallow feedforward artificial neural networks with ReLU activation. This property is one of the fundamental artifacts separating ReLU from other commonly used activation functions. We propose a kind of closure of the search space so that in the extended space minimizers exist. In a second step, we show under mild assumptions that the newly added functions in the extension perform worse than appropriate representable ReLU networks. This then implies that the optimal response in the extended target space is indeed the response of a ReLU network.
Journal Article Details
Publisher Name: Global Science Press
Language: English
DOI: https://doi.org/10.4208/jml.230903
Journal of Machine Learning, Vol. 3 (2024), Iss. 1 : pp. 1–22
Published online: 2024-01
AMS Subject Headings:
Copyright: COPYRIGHT: © Global Science Press
Pages: 22
Keywords: Neural Networks Shallow Networks Best Approximation ReLU Activation Approximatively Compact.