A Stochastic Three-Block Alternating Minimization Algorithm and Its Application to Quantized Deep Neural Networks

Abstract

Deep neural networks (DNNs) have made great progress in various fields. In particular, the quantized neural network is a promising technique for making DNNs compatible with resource-limited devices for memory and computation saving. In this paper, we mainly consider a non-convex minimization model with three blocks to train quantized DNNs and propose a novel stochastic three-block alternating minimization (STAM) algorithm to solve it. We develop a convergence theory for the STAM algorithm and obtain an $\epsilon$-stationary point with an optimal convergence rate. Furthermore, we implement our STAM algorithm to train DNNs with relaxed binary weights. The experiments are carried out on three different network structures, namely VGG-11, VGG-16, and ResNet-18. These DNNs are trained using two different datasets, CIFAR-10 and CIFAR-100, respectively. We compare our STAM algorithm with state-of-the-art algorithms for training quantized neural networks. The test accuracy indicates the effectiveness of our model and algorithm for training relaxed binary quantization DNNs.