> ML_LITERATURE // KINGMA-BA-2014-ADAM-METHOD-FOR-STOCHASTIC-OPTIMIZATION_v1.0
Adam: A Method for Stochastic Optimization
Diederik P. Kingma, Jimmy Ba · International Conference on Learning Representations (ICLR) (2014)
algorithm2014industry-standardthirdPartyReproduced
Principal Contribution
Combined adaptive learning rates from RMSProp with momentum from SGD, incorporating bias corrections for first and second moments.
Operational Relevance
The universal default optimizer across all deep learning and language model training pipelines globally.
Assumptions
- Gradient moments can be estimated online via exponentially decaying running averages; coordinates are treated quasi-independently
Limitations
- Can fail to converge to optimal flat minima in vision tasks compared to tuned SGD with momentum; requires AdamW for weight decay
Connected Algorithms, Architectures & Tools
Related Algorithms:
Related Architectures:
Implementing Libraries: