In this video, I will bring a brief theory of knowledge distillation, then I will use a Keras example to show how it is simple to be implemented.
In summary, knowledge distillation means transferring knowledge from a big model (teacher) to a small model(student). The lottery ticket hypothesis may justify why knowledge distillation can work.
original code: keras.io/examples/vision/knowledge_distillation/