Dropout: A Simple Way to Prevent Neural Networks from Overfitting

Randomly dropping units during training prevents complex co-adaptations of feature detectors, acting as an implicit ensemble of 2^N thinned networks.

Universal regularizer in deep learning and key mechanism for MC-Dropout epistemic uncertainty estimation in production.

Assumptions

• Multiplying test-time weights by retention probability p exactly averages predictions across the combinatorial space of sub-networks

Limitations

• Increases training epochs required for convergence by 2x-3x; less commonly used in modern pre-LN Transformers during pretraining