Learning Accurate Low-Bit Deep Neural Networks with Stochastic Quantization

Yinpeng Dong, Jianguo Li and Renkun Ni

Abstract

Low-bit deep neural networks (DNNs) become critical for embedded applications due to their low storage requirement and computing efficiency. However, they suffer much from the non-negligible accuracy drop. This paper proposes the stochastic quantization (SQ) algorithm for learning accurate low-bit DNNs. The motivation is due to the following observation. Existing training algorithms approximate the real-valued elements/filters with low-bit representation all together in each iteration. The quantization errors may be small for some elements/filters, while are remarkable for others, which lead to inappropriate gradient direction during training, and thus bring notable accuracy drop. Instead, SQ quantizes a portion of elements/filters to low-bit with a stochastic probability inversely proportional to the quantization error, while keeping the other portion unchanged with full-precision. The quantized and full-precision portions are updated with corresponding gradients separately in each iteration. The SQ ratio is gradually increased until the whole network is quantized. This procedure can greatly compensate the quantization error and thus yield better accuracy for low-bit DNNs.

Session

Orals - Matching

Files

PDF iconPaper (PDF)

DOI

10.5244/C.31.189
https://dx.doi.org/10.5244/C.31.189

Citation

Yinpeng Dong, Jianguo Li and Renkun Ni. Learning Accurate Low-Bit Deep Neural Networks with Stochastic Quantization. In T.K. Kim, S. Zafeiriou, G. Brostow and K. Mikolajczyk, editors, Proceedings of the British Machine Vision Conference (BMVC), pages 189.1-189.12. BMVA Press, September 2017.

Bibtex

            @inproceedings{BMVC2017_189,
                title={Learning Accurate Low-Bit Deep Neural Networks with Stochastic Quantization},
                author={Yinpeng Dong, Jianguo Li and Renkun Ni},
                year={2017},
                month={September},
                pages={189.1-189.12},
                articleno={189},
                numpages={12},
                booktitle={Proceedings of the British Machine Vision Conference (BMVC)},
                publisher={BMVA Press},
                editor={Tae-Kyun Kim, Stefanos Zafeiriou, Gabriel Brostow and Krystian Mikolajczyk},
                doi={10.5244/C.31.189},
                isbn={1-901725-60-X},
                url={https://dx.doi.org/10.5244/C.31.189}
            }