Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

Unified all NLP tasks into a common text-to-text format using an encoder-decoder Transformer pre-trained on C4 with span corruption.

Standard architecture for enterprise document summarization, extraction, question answering, and FLAN instruction-tuned checkpoints.

Assumptions

• Any NLP task can be cast as receiving text as input and generating target text as output

Limitations

• Encoder-decoder architectures require running full encoder cross-attention for each generated token, incurring higher latency than pure causal decoders