Generalization Error Analysis of Neural Networks with Gradient Based Regularization (Communications in Computational Physics, 2022)
In this work, we study gradient-based regularization methods for neuralnetworks. We mainly focus on two regularization methods: the total variation and theTikhonov regularization. Adding the regularization term to the training loss is equivalent to using neural networks to solve some variational problems, mostly in high dimensions in practical applications. We introduce a general framework to analyze theerror between neural network solutions and true solutions to variational problems.The error consists of three parts: the approximation errors of neural networks, thequadrature errors of numerical integration, and the optimization error. We also applythe proposed framework to two-layer networks to derive a priori error estimate whenthe true solution belongs to the so-called Barron space. Moreover, we conduct somenumerical experiments to show that neural networks can solve corresponding variational problems sufficiently well. The networks with gradient-based regularization aremuch more robust in image applications.
Keywords: Machine learning; regularization; generalization error; image classification.