Year: 2019
Author: Penghang Yin, Shuai Zhang, Yingyong Qi, Jack Xin
Journal of Computational Mathematics, Vol. 37 (2019), Iss. 3 : pp. 349–359
Abstract
We present LBW-Net, an efficient optimization based method for quantization and training of the low bit-width convolutional neural networks (CNNs). Specifically, we quantize the weights to zero or powers of 2 by minimizing the Euclidean distance between full-precision weights and quantized weights during backpropagation (weight learning). We characterize the combinatorial nature of the low bit-width quantization problem. For 2-bit (ternary) CNNs, the quantization of $N$ weights can be done by an exact formula in $O$($N$ log $N$) complexity. When the bit-width is 3 and above, we further propose a semi-analytical thresholding scheme with a single free parameter for quantization that is computationally inexpensive. The free parameter is further determined by network retraining and object detection tests. The LBW-Net has several desirable advantages over full-precision CNNs, including considerable memory savings, energy efficiency, and faster deployment. Our experiments on PASCAL VOC dataset [5] show that compared with its 32-bit floating-point counterpart, the performance of the 6-bit LBW-Net is nearly lossless in the object detection tasks, and can even do better in real world visual scenes, while empirically enjoying more than 4× faster deployment.
You do not have full access to this article.
Already a Subscriber? Sign in as an individual or via your institution
Journal Article Details
Publisher Name: Global Science Press
Language: English
DOI: https://doi.org/10.4208/jcm.1803-m2017-0301
Journal of Computational Mathematics, Vol. 37 (2019), Iss. 3 : pp. 349–359
Published online: 2019-01
AMS Subject Headings:
Copyright: COPYRIGHT: © Global Science Press
Pages: 11
Keywords: Quantization Low bit width deep neural networks Exact and approximate analytical formulas Network training Object detection.
Author Details
-
Convergence of a Relaxed Variable Splitting Coarse Gradient Descent Method for Learning Sparse Weight Binarized Activation Neural Network
Dinh, Thu | Xin, JackFrontiers in Applied Mathematics and Statistics, Vol. 6 (2020), Iss.
https://doi.org/10.3389/fams.2020.00013 [Citations: 1] -
Sparsity Regularization-Based Real-Time Target Recognition for Side Scan Sonar with Embedded GPU
Li, Zhuoyi | Chen, Deshan | Yip, Tsz Leung | Zhang, JinfenJournal of Marine Science and Engineering, Vol. 11 (2023), Iss. 3 P.487
https://doi.org/10.3390/jmse11030487 [Citations: 5] -
Learning quantized neural nets by coarse gradient method for nonlinear classification
Long, Ziang | Yin, Penghang | Xin, JackResearch in the Mathematical Sciences, Vol. 8 (2021), Iss. 3
https://doi.org/10.1007/s40687-021-00281-4 [Citations: 0] -
Binarized Weight Neural-Network Inspired Ultra-Low Power Speech Recognition Processor with Time-Domain Based Digital-Analog Mixed Approximate Computing
Liu, Bo | Cai, Hao | Gong, Yu | Zhu, Wentao | Li, Yan | Ge, Wei | Wang, Zhen2020 IEEE International Symposium on Circuits and Systems (ISCAS), (2020), P.1
https://doi.org/10.1109/ISCAS45731.2020.9181172 [Citations: 4] -
Gradient Descent Learning With Floats
Sun, Tao | Tang, Ke | Li, DongshengIEEE Transactions on Cybernetics, Vol. 52 (2022), Iss. 3 P.1763
https://doi.org/10.1109/TCYB.2020.2997399 [Citations: 14] -
Binary Quantized Network Training With Sharpness-Aware Minimization
Liu, Ren | Bian, Fengmiao | Zhang, XiaoqunJournal of Scientific Computing, Vol. 94 (2023), Iss. 1
https://doi.org/10.1007/s10915-022-02064-7 [Citations: 5] -
Sparse loss-aware ternarization for neural networks
Zhou, Ruizhi | Niu, Lingfeng | Xu, DachuanInformation Sciences, Vol. 693 (2025), Iss. P.121668
https://doi.org/10.1016/j.ins.2024.121668 [Citations: 0] -
An Ultra-Low Power Always-On Keyword Spotting Accelerator Using Quantized Convolutional Neural Network and Voltage-Domain Analog Switching Network-Based Approximate Computing
Liu, Bo | Wang, Zhen | Zhu, Wentao | Sun, Yuhao | Shen, Zeyu | Huang, Lepeng | Li, Yan | Gong, Yu | Ge, WeiIEEE Access, Vol. 7 (2019), Iss. P.186456
https://doi.org/10.1109/ACCESS.2019.2960948 [Citations: 20] -
DAC: Data-Free Automatic Acceleration of Convolutional Networks
Li, Xin | Zhang, Shuai | Jiang, Bolan | Qi, Yingyong | Chuah, Mooi Choo | Bi, Ning2019 IEEE Winter Conference on Applications of Computer Vision (WACV), (2019), P.1598
https://doi.org/10.1109/WACV.2019.00175 [Citations: 4] -
Advances in Information and Communication
Training with Reduced Precision of a Support Vector Machine Model for Text Classification
Żurek, Dominik | Pietroń, Marcin | Wiatr, Kazimierz2021
https://doi.org/10.1007/978-3-030-73103-8_56 [Citations: 0]