> ML_LITERATURE // LEWIS-2020-BART-DENOISING-SEQUENCE-TO-SEQUENCE-PRETRAINING_v1.0
BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, Luke Zettlemoyer · Annual Meeting of the Association for Computational Linguistics (ACL) (2020)
seminal-architecture2020industry-standardthirdPartyReproduced
Principal Contribution
Combined bidirectional encoder (BERT-style) and autoregressive decoder (GPT-style) into a denoising sequence-to-sequence autoencoder trained with arbitrary document corruptions.
Operational Relevance
Directly guides architectural decisions, alignment strategy, and serving infrastructure for task-summarization, task-text-generation.
Assumptions
- Empirical distribution regularity holds and target domain adheres to pretraining linguistic/visual support
Limitations
- Resource scaling, inference memory requirements, and alignment robustness vary with model size and hardware topology
Connected Algorithms, Architectures & Tools
Related Algorithms:
Related Architectures:
Implementing Libraries: