Skip to main content

> ML_LITERATURE // HENDRYCKS-2016-GAUSSIAN-ERROR-LINEAR-UNITS-GELU_v1.0

Gaussian Error Linear Units (GELUs)

Dan Hendrycks, Kevin Gimpel · arXiv preprint (2016)

algorithm2016industry-standardthirdPartyReproduced
arXiv:1606.08415Back to Literature

Principal Contribution

Formulated the GELU activation function multiplying inputs by the standard Gaussian cumulative distribution: x * Phi(x), smoothing the ReLU singularity.

Operational Relevance

Serves as canonical technical reference for implementing task-text-generation, task-multiclass-classification in production systems.

Assumptions

  • Underlying data distribution satisfies empirical consistency and regularity assumptions across training domains

Limitations

  • Scaling characteristics and accuracy depend on hardware architecture, parameter scale, and dataset quality

Connected Algorithms, Architectures & Tools

Related Algorithms:
alg-autoregressive-self-supervised-gptMasked Language Modeling (MLM / BERT)
Related Architectures:
Implementing Libraries: