Journals
Resources
About Us
Open Access

Enhancing Accuracy in Deep Learning Using Random Matrix Theory

Year:    2024

Author:    Leonid Berlyand, Etienne Sandier, Yitzchak Shmalo, Lei Zhang

Journal of Machine Learning, Vol. 3 (2024), Iss. 4 : pp. 347–412

Abstract

We explore the applications of random matrix theory (RMT) in the training of deep neural networks (DNNs), focusing on layer pruning that reduces the number of DNN parameters (weights). Our numerical results show that this pruning leads to a drastic reduction of parameters while not reducing the accuracy of DNNs and convolutional neural network (CNNs). Moreover, pruning the fully connected DNNs actually increases the accuracy and decreases the variance for random initializations. Our numerics indicate that this enhancement in accuracy is due to the simplification of the loss landscape. We next provide rigorous mathematical underpinning of these numerical results by proving the RMT-based Pruning Theorem. Our results offer valuable insights into the practical application of RMT for the creation of more efficient and accurate deep-learning models.

Journal Article Details

Publisher Name:    Global Science Press

Language:    English

DOI:    https://doi.org/10.4208/jml.231220

Journal of Machine Learning, Vol. 3 (2024), Iss. 4 : pp. 347–412

Published online:    2024-01

AMS Subject Headings:   

Copyright:    COPYRIGHT: © Global Science Press

Pages:    66

Keywords:    Deep learning Marchenko-Pastur distribution Random matrix theory Increasing accuracy Pruning.

Author Details

Leonid Berlyand

Etienne Sandier

Yitzchak Shmalo

Lei Zhang

  1. Stability of accuracy for the training of DNNs via the uniform doubling condition

    Shmalo, Yitzchak

    Annals of Mathematics and Artificial Intelligence, Vol. 92 (2024), Iss. 2 P.439

    https://doi.org/10.1007/s10472-023-09919-1 [Citations: 0]