Training set sufficiency for image analysis

Patent metadata

Field	Value
Publication number	US-12169984-B2
Application number	US-202117157427-A
Country	US
Kind code	B2
Filing date	Jan 25, 2021
Priority date	May 11, 2018
Publication date	Dec 17, 2024
Grant date	Dec 17, 2024

Abstract

Aspects of the technology described herein improve an object recognition system by specifying a type of picture that would improve the accuracy of the object recognition system if used to retrain the object recognition system. The technology described herein can take the form of an improvement model that improves an object recognition model by suggesting the types of training images that would improve the object recognition model's performance. For example, the improvement model could suggest that a picture of a person smiling be used to retrain the object recognition system. Once trained, the improvement model can be used to estimate a performance score for an image recognition model given the set characteristics of a set of training of images. The improvement model can then select a feature of an image, which if added to the training set, would cause a meaningful increase in the recognition system's performance.

First claim

What is claimed is:

1. A method of improving a performance of an object recognition model comprising:
training the object recognition model to recognize an object using a set of training images;
calculating a performance score measuring the object recognition model's ability to accurately identify the object in one or more validation images;
selecting an image characteristic that would cause a performance improvement in the object recognition model when a new image containing the image characteristic is added to a new set of training images;
upon selecting the image characteristic, outputting for display to a user an interface asking the user to generate a new training image with a label by selecting an area of the new training image that depicts the object associated with the label, wherein the user selects the new training image from a set of new training images including the image characteristic displayed in a user interface;
adding the new training image to the new set of training images; and
retraining the object recognition model using the new set of training images.

2. The method of claim 1 , wherein the method further comprises:
generating image characteristics for each image in the set of training images;
generating set characteristics for the set of training images by analyzing the image characteristics of each image in the set of training images, wherein the set characteristics describe characteristics of the set of training images;
associating the performance score with the set characteristics to generate an improvement model training set;
using the improvement model training set to train an improvement model; and
using the improvement model to select the image characteristic.

3. The method of claim 2 , wherein the set characteristics comprise a coefficient of variance for the image characteristics of images in the set of training images.

4. The method of claim 2 , wherein the improvement model is a random decision forest model.

5. The method of claim 2 , wherein the method further comprises generating a new set of characteristics for the new set of training images, and wherein the improvement model uses the new set of characteristics to select the image characteristic.

6. The method of claim 5 , wherein the method further comprises calculating a performance measure of the object recognition model using the new set of characteristics as input to the improvement model.

7. The method of claim 6 , wherein the improvement model training set includes pairs of set characteristic values and the performance measure.

8. A method for improving a performance of an object recognition model comprising:
training an object recognition model to recognize a first person using a set of training images;
calculating a performance score for the set of training images using validation images of the first person that were not included in the set of training images, the performance score measuring the object recognition model's ability to accurately identify the first person in the validation images;
selecting an image characteristic that would cause a performance improvement in the object recognition model when a new image containing the image characteristic is added to a new set of training images;
upon selecting the image characteristic, outputting for display to a user interface a request that the new image containing the image characteristic be labeled, wherein the image characteristic includes a feature of the first person extracted by the object recognition model;
receiving from the user interface a label for the new image; and
adding the new image and the label to the new set of training images.

9. The method of claim 8 , further comprising generating set characteristics for the set of training images by analyzing image characteristics of images in the set of training images, wherein the set characteristics describe characteristics of the set of training images as a whole.

10. The method of claim 9 , wherein the method further comprises:
associating the performance score with the set characteristics to generate an improvement model training set;
using the improvement model training set to train an improvement model;
receiving the new set of training images for the object recognition model, the new set of training images depicting a second person different from the first person; and
using the improvement model to determine a predicted performance measure of the object recognition model after training with the new set of training images.

11. The method of claim 10 , wherein the method further comprises training the object recognition model to recognize the second person using the new set of training images and the new image.

12. The method of claim 9 , wherein a set characteristic comprises a coefficient of variance for smile intensity of images in the set of training images.

13. The method of claim 9 , wherein a set characteristic comprises a coefficient of variance for exposure of images in the set of training images.

14. The method of claim 9 , wherein a set characteristic comprises a coefficient of variance for a facial landmark in images in the set of training images.

15. The method of claim 9 , wherein the image characteristic is limited to characteristics that are identifiable to a user looking at an image.

16. The method of claim 8 , wherein the feature of the object extracted by the object recognition model includes at least one of:
a land mark;
a point of interest; and
image meta data.

17. A computer-storage media having computer-executable instructions embodied thereon that when executed by a computer processor cause a computing device to perform a method, the method comprising:
receiving a new set of training images for a facial recognition model, the new set of training images depicting a first person;
selecting an image characteristic that would cause a performance improvement in the facial recognition model when a new image containing the image characteristic is added to the new set of training images;
upon selecting the image characteristic, outputting for display to a user an interface asking the user to select a new training image that depicts the image characteristic of the first person;
in response to receive a selecting of the new training image by the user, adding the new training image to the new set of training images; and
retraining the facial recognition model using the new set of training images.

18. The media of claim 17 , wherein the method further comprises:
generating image characteristics for each image in a set of training images;
generating set characteristics for the set of training images by analyzing the image characteristics of each image in the set of training images, wherein the set characteristics describe characteristics of the set of training images;
training the facial recognition model to recognize a face using the set of training images;
calculating a performance score for the set of training images, the performance score measuring the facial recognition model's ability to accurately identify the face in one or more validation images;
associating the performance score with the set characteristics to generate an improvement model training set;
using the improvement model training set to train a random decision forest model; and
using the random decision forest model to identify the image characteristic.

19. The media of claim 18 , wherein the improvement model training set includes pairs of set characteristic values and the performance score.

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Nir Oron
• Ronen Royi
• Jassin Ohad
• Gada Milan M
• Pipek Mor Geva

Classifications

• G06V10/776

  Validation; Performance evaluation · CPC title

• G06V10/774

  Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06V10/70 Primary

  using pattern recognition or machine learning (optical pattern recognition or electronic computations therefor G06V10/88) · CPC title

• G06F18/217

  Validation; Performance evaluation; Active pattern learning techniques · CPC title

• G06F18/211

  Selection of the most significant subset of features · CPC title