Enhanced Expressive Power and Fast Training of Neural Networks by Random Projections (CSIAM Transactions on Applied Mathematics, 2021)
Random projections are able to perform dimension reduction efficiently fordatasets with nonlinear low-dimensional structures. One well-known example is thatrandom matrices embed sparse vectors into a low-dimensional subspace nearly isometrically, known as the restricted isometric property in compressed sensing. In thispaper, we explore some applications of random projections in deep neural networks.We provide the expressive power of fully connected neural networks when the inputdata are sparse vectors or form a low-dimensional smooth manifold. We prove that thenumber of neurons required for approximating a Lipschitz function with a prescribedprecision depends on the sparsity or the dimension of the manifold and weakly on thedimension of the input vector. The key in our proof is that random projections embed stably the set of sparse vectors or a low-dimensional smooth manifold into a low-dimensional subspace. Based on this fact, we also propose some new neural networkmodels, where at each layer the input is first projected onto a low-dimensional subspace by a random projection and then the standard linear connection and non-linearactivation are applied. In this way, the number of parameters in neural networks issignificantly reduced, and therefore the training of neural networks can be acceleratedwithout too much performance loss.
Keywords: Neural network; approximation; random projection.