First-Order and Second-Order Pruning

First-Order and Second-Order Pruning

Magnitude pruning is the simplest type of unstructured pruning, which only considers the current value of the weight during training. It is called a “first-order” pruning method because it doesn't consider the prior values or how they are changing.

Movement pruning is a generalization of magnitude pruning, which considers the “rate of change” of the weight, as it changes with each training step. This is called a “second-order” pruning method because it is considering the “gradient” or “derivative” in the mathematical sense.

The slope of the curve helps decide whether to prune a weight. For example, if a weight has a magnitude below a threshold and its rate of change is decreasing so that it is moving towards zero, then it is pruned to zero. In this way, tiny weights that are still increasing in magnitude, whether positive or negative, are not yet pruned.