 Hello, and welcome to the presentation of Learned Step-Size Quantization, or LSQ. I'm Steve Essar, and this is work that was done with my colleagues at IBM Research. So quantized neural networks are becoming increasingly popular as a means to decrease memory footprint and increase throughput in resource-constrained settings. With this interest, there is increasing importance to the question of how to quantize networks typically from 32-bit to 8-bit or less, while still preserving network task performance. Our approach, LSQ, is a method to determine the step size or bandwidth of the quantizer for each weight and or activation layer in a network. We consider the case of a standard uniform quantizer that maps unquantized values to quantized values equally spaced by the step size. So determining the step size is critical as if the value is too small or too large, the resulting quantized data will provide a poor representation of the original data, in turn leading to poor application performance. So to date, a number of methods have been used for quantizing neural networks, and the means in which they determine step size can be divided into the four main categories you see here. LSQ falls into the fourth category, using back propagation to learn the step size as a parameter during standard training. LSQ innovates in this space by improving the approximation of the step size loss gradient and by providing a principal means to scale this gradient. So strictly speaking, the derivative through a standard uniform quantizer is zero for almost all values due to the presence of the round function. You can note this in the staircase shape of the quantizer that is flat at almost all locations except the transition points. So LSQ computes the derivative through the quantizer to the step size parameter by treating the round function as a pass-through function for the purposes of differentiation. And this is similar to an approach commonly taken in quantized networks for computing the derivative through the quantizer to its input. And as seen on the right, in contrast to the derivative computed by other approaches that also learn the step size, the approach of LSQ provides a derivative that is sensitive to the quantizer transition points. Thus, if a value to quantize is near a transition point and therefore requires only a small change in step size to move to the next quantized state, it produces a higher derivative than a value that is further from a transition point and thus requires a large change in step size before it moves to the next quantized bin. So in addition to this approximation, LSQ provides a principal means to scale the step size loss gradient. So in learning with gradient descent, it's well known that if a parameter update is large, too large or too small relative to the magnitude of the parameter being updated, then poor convergence will result. We found that the magnitude of the step size update scales with the size of the population to be quantized and the bits of precision leading to updates that are much larger than those made to network weights under most conditions and in turn poor convergence. We provide a simple correction for this imbalance by rescaling the step size loss gradient based on population size and precision leading to an overall improvement in network task accuracy. I'll conclude now with our main result where we show that on the ImageNet dataset, LSQ achieves better performance than all prior quantization methods for the networks considered with an improvement of over 1% in some cases. We also found that when combined with self-knowledge distillation using a technique previously demonstrated for quantized networks by Nishra and Maher, the networks quantized to as little as three bits of precision were able to match the accuracy of full-precision equivalent networks the first time that this has been demonstrated. LSQ is easy to implement as part of a standard training flow and for those so interested it would provide pseudocode in the appendix of our paper. The paper also contains further details on the methods and additional results that show that LSQ does not actually minimize quantization error despite outperforming techniques that do and we also demonstrate an advantage to using two-bit networks in certain cases. Well, thank you for your time.