Parameter Removal: Identify the trainable layers in the model and reduce the number of parameters!
Weight Pruning: Specific connections between neurons are removed. In practice this removal means replacing the weight by zero. Will this removal not affect the knowledge learned by the model? Do this removal until it does not decrease the accuracy much (acceptable level).
Optimal Brain Damage: Yann LeCun et. al   proposed the following:
Train a network.
Estimate the importance of each weight by watching how the loss would change upon perturbing the weight. Smaller change means less importance. (This importance is called the saliency.)
Remove the weights with low importance.
Go back to Step 1. and retrain the network, permanently fixing the removed weights to zero.