Systems and methods for automatically annotating images

Patent metadata

Field	Value
Publication number	US-11720623-B2
Application number	US-202017097601-A
Country	US
Kind code	B2
Filing date	Nov 13, 2020
Priority date	Nov 14, 2019
Publication date	Aug 8, 2023
Grant date	Aug 8, 2023

Abstract

In some embodiments, apparatuses and methods are provided herein useful to automatically annotating images. In some embodiments, a system for automatically annotating images comprises a database, wherein the database is configured to store images and annotations for the images and a control circuit, wherein the control circuit is communicatively coupled to the database, and wherein the control circuit is configured to retrieve, from the database, an image, generate, based on the image, a collection of augmented images, generate segmentation maps for each image in the collection of augmented images, wherein each of the segmentation maps include segments, select, based on a threshold, ones of the segments above a threshold, merge the ones of the segments above the threshold to create a segmented image, and generate, for each segment of the segmented image, classifications, wherein an annotation for the image includes the segmented images and the classifications.

First claim

What is claimed is:

1. A system for automatically annotating images, the system comprising:
a database, wherein the database is configured to store images and annotations for the images; and
a control circuit, wherein the control circuit is communicatively coupled to the database, and wherein the control circuit is configured to:
retrieve, from that database, an image;
generate, based on the image, a collection of augmented images;
generate segmentation maps for each image in the collection of augmented images, wherein each of the segmentation maps include segments;
select, based on a threshold, ones of the segments above a threshold;
merge the ones of the segments above the threshold to create a segmented image; and
generate, for each segment of the segmented image, classifications, wherein an annotation for the image includes the segmented images and the classifications.

2. The system of claim 1 , further comprising:
an image capture device, wherein the image capture device is configured to capture images of carts.

3. The system of claim 2 , wherein the control circuit is further configured to:
identify, based on a machine learning model, the images, and the annotations for the images, products located in the carts.

4. The system of claim 3 , wherein the control circuit is further configured to:
compile, based on transaction histories associated with the carts, a list of products purchased with each cart, wherein the identification of the products located in the carts is further based on the list of products purchased with each cart.

5. The system of claim 2 , wherein the control circuit is further configured to:
segment the images of carts.

6. The system of claim 1 , wherein the control circuit generates the augmented images by one or more of adjusting color of the image, adjusting noise of the image, adjusting sharpness of the image, rotating the image, and cropping the image.

7. The system of claim 1 , wherein the control circuit generates the augmented images randomly.

8. The system of claim 1 , wherein the control circuit generates at least twenty augmented images based on the image.

9. The system of claim 1 , wherein the threshold is a consistency threshold representing a consistency of the segments between the segmentation maps.

10. The system of claim 9 , wherein the consistency threshold is at least 70%.

11. A method for automatically annotating images, the method comprising:
storing, in a database, images and annotations for the images;
retrieving, by a control circuit from the database, an image;
generating, by the control circuit based on the image, a collection of augmented images;
generating, by the control circuit, segmentation maps for each of the augmented images, wherein each of the segmentation maps includes segments;
selecting, by the control circuit based on a threshold, ones of the segments above the threshold;
merging, by the control circuit, the ones of the segments above the threshold to create a segmented image; and
generating, for each segment of the segmented image, classifications, wherein an annotation for the image includes the segmented image and the classifications.

12. The method of claim 11 , further comprising:
capturing, by an image capture device, images of carts.

13. The method of claim 12 , further comprising:
identifying, by the control circuit based on a machine learning model, the images, and the annotations for the images, products located in the carts.

14. The method of claim 13 , further comprising:
compiling, by the control circuit based on transaction histories associated with the carts, a list of products purchased with each cart, wherein the identifying the products located in the carts is further based on the list of products purchased with each cart.

15. The method of claim 12 , further comprising:
segmenting, by the control circuit, the images of the carts.

16. The method of claim 11 , wherein the control circuit generates the augmented images by one or more of adjusting color of the image, adjusting noise of the image, adjusting sharpness of the image, rotating the image, and cropping the image.

17. The method of claim 11 , wherein the control circuit generates the augments images randomly.

18. The method of claim 11 , wherein the control circuit generates at least twenty augmented images.

19. The method of claim 11 , wherein the threshold is a consistency threshold representing a consistency of the segments between the segmentation maps.

20. The method of claim 19 , wherein the consistency threshold is at least 70%.

Assignees

• Walmart Apollo Llc

Inventors

• Liu Haining
• Zhu Feiyun
• Reagan Ryan B
• Xiao Zhichun

Classifications

• G06T3/10

  Selection of transformation methods according to the characteristics of the input images · CPC title

• G06F16/55 Primary

  Clustering; Classification · CPC title

• G06F18/2148

  characterised by the process organisation or structure, e.g. boosting cascade · CPC title

• G06N20/00

  Machine learning · CPC title

• G06T3/0056

  Physics · mapped topic