Scalable knowledge distillation techniques for machine learning

Patent metadata

Field	Value
Publication number	US-12307750-B2
Application number	US-202217837636-A
Country	US
Kind code	B2
Filing date	Jun 10, 2022
Priority date	Jun 10, 2022
Publication date	May 20, 2025
Grant date	May 20, 2025

Abstract

A data processing system implements a dynamic knowledge distillation process including dividing training data into a plurality of batches of samples and distilling a student model from a teacher model using an iterative knowledge distillation. The process includes instantiating an instance of the teacher model and the student model in a memory of the data processing system and obtaining a respective batch of training data from the plurality of batches of samples in the memory. The process includes training the teacher and student models using each of the samples in the respective batch of the training data, evaluating the performance of the student model compared with the performance of the teacher model, and providing feedback to student model to adjust the behavior of the student model based on the performance of the student model.

First claim

What is claimed is:

1. A data processing system comprising:
a processor; and
a machine-readable storage medium storing executable instructions that, when executed, cause the processor to perform operations comprising:
instantiating an instance of a teacher model in a memory of the data processing system;
instantiating an instance of a student model in the memory of the data processing system;
dividing training data into a plurality of batches of samples, wherein a size of the plurality of batches of samples is based at least in part on an amount of the memory available for training the student model after instantiating the instance of the teacher model and the instance of the student model; and
training the student model to replicate performance of the teacher model using an iterative knowledge distillation process in which the teacher model and the student model are trained in parallel during each iteration of the iterative knowledge distillation process by:
obtaining a respective batch of training data from the plurality of batches of samples in the memory;
training the teacher model using each of the samples in the respective batch of training data;
training the student model using each of the samples in the respective batch of training data;
evaluating performance of the student model compared with the performance of the teacher model; and
providing feedback to student model to adjust behavior of the student model based on the performance of the student model.

2. The data processing system of claim 1 , for each respective batch of the plurality of batches of samples that have not yet been processed, performing operations of:
training the teacher model using each of the samples in the respective batch of training data;
training the student model using each of the samples in the respective batch of training data;
evaluating the performance of the student model; and
providing feedback to student model based on the performance of the student model.

3. The data processing system of claim 1 , wherein the machine-readable storage medium includes instructions configured to cause the processor to perform operations of:
comparing first logits associated with the teacher model with second logits associated with the student model; and
determining the feedback to the student model based on differences between the first logits and the second logits.

4. The data processing system of claim 1 , wherein the machine-readable storage medium includes instructions configured to cause the processor to perform operations of:
comparing first labels output by the teacher model with second labels output by the student model; and
determining the feedback to the student model based on differences between the first labels and the second labels.

5. The data processing system of claim 1 , wherein the machine-readable storage medium includes instructions configured to cause the processor to perform operations of:
filtering the training data used to train the student model to exclude training data not associated with features for which the student model is to be trained.

6. The data processing system of claim 5 , wherein the machine-readable storage medium includes instructions configured to cause the processor to perform operations of:
adjusting weights associated with first logits associated with the teacher model to reflect an importance of data classes based on a likelihood of the student model receiving input data associated with the data classes to obtain first weighted logits;
comparing the first weighted logits with second logits associated with the student model; and
determining the feedback to the student model based on differences between the first weighted logits and the second logits.

7. The data processing system of claim 5 , wherein the machine-readable storage medium includes instructions configured to cause the processor to perform operations of:
pretraining the teacher model using all the training data prior to distilling the student model from the teacher model.

8. A method implemented in a data processing system for distilling a student model from a teacher model, the method comprising:
instantiating an instance of a teacher model in a memory of the data processing system;
instantiating an instance of a student model in the memory of the data processing system;
dividing training data into a plurality of batches of samples, wherein a size of the plurality of batches of samples is based at least in part on an amount of the memory available for training the student model after instantiating the instance of the teacher model and the instance of the student model; and
training the student model to replicate performance of the teacher model using an iterative knowledge distillation process in which the teacher model and the student model are trained in parallel during each iteration of the iterative knowledge distillation process by:
obtaining a respective batch of training data from the plurality of batches of samples in the memory;
training the teacher model using each of the samples in the respective batch of training data;
training the student model using each of the samples in the respective batch of training data;
evaluating performance of the student model compared with the performance of the teacher model; and
providing feedback to student model to adjust behavior of the student model based on the performance of the student model.

9. The method of claim 8 , the method further comprising for each respective batch of the plurality of batches of samples that have not yet been processed:
training the teacher model using each of the samples in the respective batch of training data;
training the student model using each of the samples in the respective batch of training data;
evaluating the performance of the student model; and
providing feedback to student model based on the performance of the student model.

10. The method of claim 8 , further comprising:
comparing first logits associated with the teacher model with second logits associated with the student model; and
determining the feedback to the student model based on differences between the first logits and the second logits.

11. The method of claim 8 , further comprising:
comparing first labels output by the teacher model with second labels output by the student model; and
determining the feedback to the student model based on differences between the first labels and the second labels.

12. The method of claim 8 , further comprising:
filtering the training data used to train the student model to exclude training data not associated with features for which the student model is to be trained.

13. The method of claim 11 , further comprising:
adjusting weights associated with first logits associated with the teacher model to reflect an importance of data classes based on a likelihood of the student model receiving input data associated with the data classes to obtain first weighted logits;
comparing the first weighted logits with second logits associated with the student model; and
determining the feedback to the student model based on differences between the first weighted logits and the second logits.

14. The method of claim 11 , further comprising:
pretraining the teacher model using all the training data prior to distilling the student model from the teacher model.

15. A machine-readable medium on which are stored instructions that, when executed, cause a processor of a programmable device to perform operations of:
instantiating an instance of a teacher model in a memory of the programmable device;

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Krishnan Adit
• Li Ji
• Wei Yixuan
• Yu Xiaozhi
• Hu Han
• Dai Qi

Classifications

• G06N3/045 Primary

  Combinations of networks · CPC title

• G06V10/82

  using neural networks · CPC title

• G06V10/7747

  Organisation of the process, e.g. bagging or boosting · CPC title

• G06N3/096

  Transfer learning · CPC title

• G06V10/776 Primary

  Validation; Performance evaluation · CPC title