Magnitude Pruning

Magnitude Pruning

Magnitude pruning is zeroing weights that have a small magnitude, which means they have a small numeric absolute value. Equivalently, it is the removal of near-zero weights (positive and negative).

Magnitude pruning is the simplest type of unstructured pruning. In its pure form, any of the weights in the whole model may be pruned, regardless of what structure they are in. This can be combined with structured pruning by limiting to particular structural units of the model.

Magnitude pruning can be performed as part of training, or after training. Post-training magnitude pruning is conceptually similar to quantization, in that a new model with changed weights can be created. Sometimes post-pruning re-training may be required, or it also may be avoided.