Clustering historical images using a convolutional neural net and labeled data bootstrapping

Patent metadata

Field	Value
Publication number	US-10318846-B2
Application number	US-201615393008-A
Country	US
Kind code	B2
Filing date	Dec 28, 2016
Priority date	Dec 28, 2016
Publication date	Jun 11, 2019
Grant date	Jun 11, 2019

Abstract

Systems and methods for classifying historical images. A feature extractor may create feature vectors corresponding to a plurality of images. A first classification of the plurality of images may be performed based on the plurality of feature vectors, which may include assigning a label to each of the plurality of images and assigning a probability for each of the assigned labels. The assigned probability for each of the assigned labels may be related to a statistical confidence that a particular assigned label is correctly assigned to a particular image. A subset of the plurality of images may be displayed to a display device. An input corresponding to replacement of an incorrect label with a corrected label for a certain image may be received from a user. A second classification of the plurality of images based on the input from the user may be performed.

First claim

What is claimed is:

1. A method for classifying a plurality of images comprising:
creating, by a feature extractor, a plurality of feature vectors corresponding to the plurality of images;
performing, by a feature classifier, a first classification of the plurality of images based on the plurality of feature vectors, wherein performing the first classification includes:
assigning at least one of a plurality of labels to each of the plurality of images; and
assigning a first probability for each of the assigned labels, wherein the assigned first probability for each of the assigned labels is related to a statistical confidence that a particular assigned label is correctly assigned to a particular image;

determining a subset of probabilities of the assigned first probabilities, wherein the subset of probabilities includes all of the assigned first probabilities that are less than an upper probability threshold;
determining a subset of the plurality of images corresponding to the subset of probabilities;
outputting, to a display device, the subset of the plurality of images corresponding to the subset of probabilities;
receiving user input corresponding to replacement of an incorrect label with a corrected label for a certain image of the subset of the plurality of images;
receiving user input corresponding to a confidence level associated with the corrected label;
adjusting the feature classifier using the corrected label and the confidence level associated with the corrected label; and
performing, by the adjusted feature classifier, a second classification of the plurality of images based on the plurality of feature vectors, wherein performing the second classification includes:
assigning at least one of the plurality of labels to each of the plurality of images, including assigning the corrected label to the certain image; and
assigning a second probability for each of the assigned labels.

2. The method of claim 1 , wherein the feature extractor is a convolutional neural network (CNN), the CNN having been previously trained and the CNN being compatible with the plurality of images such that the plurality of images are receivable as inputs by the CNN.

3. The method of claim 1 , wherein the plurality of images are historical images.

4. The method of claim 1 , further comprising:
determining a second subset of probabilities of the assigned second probabilities;
determining a second subset of the plurality of images corresponding to the second subset of probabilities;
outputting, to the display device, the second subset of the plurality of images; and
receiving user input corresponding to replacement of a second incorrect label with a second corrected label for a second certain image of the second subset of the plurality of images.

5. The method of claim 1 , wherein each of the plurality of feature vectors comprise 4096 numbers.

6. The method of claim 1 , wherein the subset of probabilities includes all of the
assigned first probabilities that are greater than a lower probability threshold and less than the upper probability threshold.

7. The method of claim 1 , further comprising:
receiving user input corresponding to creation of a new label, wherein the new label is added to the plurality of labels.

8. A computer readable storage media comprising instructions to cause one or more processors to perform operations comprising:
creating, by a feature extractor, a plurality of feature vectors corresponding to the plurality of images;
performing, by a feature classifier, a first classification of the plurality of images based on the plurality of feature vectors, wherein performing the first classification includes:
assigning at least one of a plurality of labels to each of the plurality of images; and
assigning a first probability for each of the assigned labels, wherein the assigned first probability for each of the assigned labels is related to a statistical confidence that a particular assigned label is correctly assigned to a particular image;

determining a subset of probabilities of the assigned first probabilities, wherein the subset of probabilities includes all of the assigned first probabilities that are less than an upper probability threshold;
determining a subset of the plurality of images corresponding to the subset of probabilities;
outputting, to a display device, the subset of the plurality of images corresponding to the subset of probabilities;
receiving user input corresponding to replacement of an incorrect label with a corrected label for a certain image of the subset of the plurality of images;
receiving user input corresponding to a confidence level associated with the corrected label;
adjusting the feature classifier using the corrected label and the confidence level associated with the corrected label; and
performing, by the adjusted feature classifier, a second classification of the plurality of images based on the plurality of feature vectors, wherein performing the second classification includes:
assigning at least one of the plurality of labels to each of the plurality of images, including assigning the corrected label to the certain image; and
assigning a second probability for each of the assigned labels.

9. The computer readable storage media of claim 8 , wherein the feature extractor is a convolutional neural network (CNN), the CNN having been previously trained and the CNN being compatible with the plurality of images such that the plurality of images are receivable as inputs by the CNN.

10. The computer readable storage media of claim 8 , wherein the plurality of images are historical images.

11. The computer readable storage media of claim 8 , further comprising instructions to cause one or more processors to perform operations further comprising:
determining a second subset of probabilities of the assigned second probabilities;
determining a second subset of the plurality of images corresponding to the second subset of probabilities;
outputting, to the display device, the second subset of the plurality of images; and
receiving user input corresponding to replacement of a second incorrect label with a second corrected label for a second certain image of the second subset of the plurality of images.

12. The computer readable storage media of claim 8 , wherein each of the plurality of feature vectors comprise 4096 numbers.

13. The computer readable storage media of claim 8 , wherein the subset of probabilities includes all of the
assigned first probabilities that are greater than a lower probability threshold and less than the upper probability threshold.

14. The computer readable storage media of claim 8 , further comprising instructions to cause one or more processors to perform operations further comprising:
receiving user input corresponding to creation of a new label, wherein the new label is added to the plurality of labels.

15. A system for classifying a plurality of images, the system comprising:
one or more processors;
a display device in communication with the one or more processors;
one or more computer readable storage mediums comprising instructions to cause the one or more processors to perform operations comprising:
creating, by a feature extractor, a plurality of feature vectors corresponding to the plurality of images;
performing, by the feature classifier, a first classification of the plurality of images based on the plurality of feature vectors, wherein performing the first classification includes:
assigning at least one of a plurality of labels to each of the plurality of images; and
assigning a first prob

Assignees

• Ancestry Com Operations Inc

Inventors

• Brown Laryn
• Murdock Michael
• Reese Jack
• Reid Shawn

Classifications

• G06V10/7788

  the supervisor being a human, e.g. interactive learning with a human teacher · CPC title

• G06V10/82 Primary

  using neural networks · CPC title

• G06F18/41

  Interactive pattern learning with a human teacher · CPC title

• G06K9/6254 Primary

  Physics · mapped topic