Training data-efficient image transformers & distillation through attention (DeiT)

Trained Vision Transformers purely on ImageNet-1K without massive JFT pre-training by introducing a distillation token interacting with class tokens via self-attention.

Directly guides architectural decisions, alignment strategy, and serving infrastructure for task-image-classification.

Assumptions

• Empirical distribution regularity holds and target domain adheres to pretraining linguistic/visual support

Limitations

• Resource scaling, inference memory requirements, and alignment robustness vary with model size and hardware topology