Model training using incomplete indications of types of defects present in training images

Patent metadata

Field	Value
Publication number	US-11250552-B1
Application number	US-202016882270-A
Country	US
Kind code	B1
Filing date	May 22, 2020
Priority date	May 31, 2018
Publication date	Feb 15, 2022
Grant date	Feb 15, 2022

Abstract

Machine learning techniques are disclosed for training a model to identify each of multiple different classes in images, based on training data where each training image may not be labeled in a complete manner with respect to the classes. The disclosed training techniques use a new label value to indicate when a ground truth value is unknown for a particular class, and do not penalize the machine learning model for output predictions that do not match the label value representing unknown ground truth. The disclosed processes may, for example, be used to train a model to detect each multiple types of image defects based on incomplete information provided by human reviewers who accept and reject images based on whether any of the types of image defects are found.

First claim

What is claimed is:

1. A system, comprising:
a data repository that stores a plurality of images with incomplete indications of which of a plurality of types of defects are present, said incomplete indications provided by human reviewers that accept and reject images based on whether the images include any of the plurality of types of defects; and
a computing system comprising one or more processors, the computing system programmed with executable instructions to use the data repository to train a machine learning model according to a process that comprises:
labeling the plurality of images based on the incomplete indications, wherein labeling the plurality of images comprises, for a first image rejected for having a first of the plurality of defect types, labeling the first image with a positive label for the first defect type and with an unknown label for each additional defect type of the plurality of defect types, each unknown label indicating that the respective type of defect may or may not be present; and
training the machine learning model with the plurality of labeled images to detect each of the plurality of defect types, wherein training the machine learning model comprises, based on said positive and unknown labels of the first image, treating the first image as a positive sample for the first defect type but not as a negative sample for any of the additional defect types.

2. The system of claim 1 , wherein training the machine learning model comprises, based the positive and unknown labels of the first image, updating parameters of the model differently for the first defect type than for the additional defect types.

3. The system of claim 1 , wherein labeling the plurality of images further comprises, for a second image accepted by a human reviewer, labeling the second image with a negative label for each of the plurality of defect types, to thereby cause the second image to be used as a negative sample for each of the plurality of defect types during said training of the machine learning model.

4. The system of claim 1 , wherein training the machine learning model comprises updating parameters of the model using a loss function that measures a difference between a prediction made by the machine learning model and an expected value.

5. The system of claim 4 , wherein the loss function prevents the machine learning model from being penalized for making an erroneous prediction for a defect type having an unknown label.

6. The system of claim 4 , wherein the loss function is responsive to the first image having the unknown label for a particular defect type by refraining from using any information associated with the particular defect type to supervise training of the model.

7. The system of claim 1 , wherein the machine learning model is a neural network, and wherein training the machine learning model comprises using the positive and unknown labels to determine how to update weights of the neural network.

8. A computer-implemented process, comprising, by execution of program instructions by a computing system:
receiving a plurality of images that have been reviewed by human reviewers who accept and reject images based on whether the images include any of a plurality of types of defects, wherein rejected images include indications of one or more types of defects found;
creating a training dataset of labeled images for training a machine learning model to detect each of the plurality of types of defects, wherein creating the training dataset comprises:
for a first image accepted by a human reviewer, labeling the first image with a negative label for each of the plurality of defect types; and
for a second image rejected by a human reviewer for including a first of the plurality of defect types, labeling the second image with a positive label for the first defect type and with an unknown label for each additional defect type of the plurality of defect types, each unknown label indicating that the respective type of defect may or may not be present; and

training the machine learning model with the training dataset, wherein training the machine learning model comprises:
based on the negative labels of the first image, using the first image as a negative sample for each of the plurality of defect types; and
based on the positive and unknown labels of the second image, using the second image as a positive sample for the first defect type but not as a negative sample for any of the additional defect types.

9. The process of claim 8 , wherein training the machine learning model comprises, based the positive and unknown labels of the second image, updating parameters of the model differently for the first defect type than for each additional defect type.

10. The process of claim 8 , wherein training the machine learning model comprises updating parameters of the model using a loss function that measures a difference between a prediction made by the machine learning model and an expected value.

11. The process of claim 10 , wherein the loss function prevents the machine learning model from being penalized for making an erroneous prediction for a defect type having an unknown label.

12. The process of claim 10 , wherein the loss function is responsive to an image having an unknown label for a particular defect type by refraining from using any information associated with the particular defect type to supervise training of the model.

13. The process of claim 10 , wherein the loss function uses the unknown labels to determine how to update parameters of the model.

14. The process of claim 8 , wherein the machine learning model is a neural network, and training the neural network comprises using the positive, negative and unknown labels to determine how to update weights of the neural network.

15. Non-transitory computer storage comprising program instructions that instruct a computing system to perform a process that comprises:
accessing a data repository that stores a plurality of images with incomplete indications of which of a plurality of types of defects are present, the incomplete indications provided by human reviewers that accept and reject images based on whether the images include any of the plurality of types of defects; and
labeling the plurality of images based on the incomplete indications, wherein labeling the plurality of images comprises, for a first image rejected for having a first of the plurality of defect types, labeling the first image with a positive label for the first defect type and with an unknown label for each additional defect type of the plurality of defect types, each unknown label indicating that the respective type of defect may or may not be present; and
training the machine learning model with the plurality of labeled images to detect each of the plurality of defect types, wherein training the machine learning model comprises, based on said positive and unknown labels of the first image, treating the first image as a positive sample for the first defect type but not as a negative sample for any of the additional defect types.

16. The non-transitory computer storage of claim 15 , wherein the executable program instructions further instruct the computing system to, based the positive and unknown labels of the first image, update parameters of the model differently for the first defect type than for the additional defect types.

17. The non-transitory computer storage of claim 15 , wherein the executable program instructions further instruct the computing system to label a second image accepted by a human reviewer with a negative label for each

Assignees

• Amazon Tech Inc

Inventors

• Padfield Dirk
• Kumar Suren

Classifications

• G06V10/82

  using neural networks · CPC title

• G06V10/454

  Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN] · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06F18/40

  Software arrangements specially adapted for pattern recognition, e.g. user interfaces or toolboxes therefor · CPC title

• G06N3/048

  Activation functions · CPC title