The optimal training recipe for knowledge distillation is consistency and patience. Consistency refers to showing the teacher and the student the exact same view of an image and additionally improving the support of the distribution with the MixUp augmentation. Patience refers to enduring long training schedules. Exciting to see advances in model compression to make stronger models more widely used!
Paper Links:
Knowledge Distillation: A Good Teacher is Patient and Consistent: arxiv.org/abs/2106.05237
Does Knowledge Distillation Really Work? arxiv.org/pdf/2106.05945.pdf
Meta Pseudo Labels: arxiv.org/pdf/2003.10580.pdf
MixUp Augmentation: keras.io/examples/vision/mixup/
Scaling Vision Transformers: arxiv.org/pdf/2106.04560.pdf
Well-Read Students Learn Better: arxiv.org/pdf/1908.08962.pdf
Chapters
0:00 Paper Title
0:05 Model Compression
1:11 Limitations of Pruning
2:13 Consistency in Distillation
4:08 Comparison with Meta Pseudo Labels
5:10 MixUp Augmentation
6:52 Patience in Distillation
8:53 Results
10:37 Exploring Knowledge Distillation
Thanks for watching! Please Subscribe!