Systems and methods for trigger-based updates to camograms for autonomous checkout in a cashier-less shopping

Patent metadata

Field	Value
Publication number	US-12373971-B2
Application number	US-202217940980-A
Country	US
Kind code	B2
Filing date	Sep 8, 2022
Priority date	Sep 8, 2021
Publication date	Jul 29, 2025
Grant date	Jul 29, 2025

Abstract

Systems and methods for tracking inventory items in an area of real space are disclosed. The method includes receiving a signal generated in dependence on sensors. The signal indicates a change to a portion of an image of an area of real space. The method includes, in response to receiving the signal, implementing a trained location detection model to determine, based on inputs, whether an inventory item identified in the portion of the image has changed a position in the area of real space. The method includes implementing a trained item classification model to determine a classification of the inventory item. The method includes updating an inventory database with inventory item data determined in dependence on the classification of the inventory item to provide an updated map of the area of real space as a result of the received signal indicating the change to the portion of the image.

First claim

What is claimed is:

1. A method for tracking inventory items in an area of real space, the method comprising:
in response to receiving a signal, generated in dependence on sensors and indicating a change to a portion of an image of an area of real space, implementing a trained location detection model to determine that an inventory item identified in the portion of the image has changed a position within a same location in the area of real space as a result of the inventory item being a shifted inventory item;
implementing a trained item classification model to determine a classification of the inventory item; and
updating an inventory database with inventory item data determined in dependence on the determined classification of the inventory item to provide an updated map of the area of real space as a result of the received signal indicating the change to the portion of the image,
wherein the trained location detection model determines that the inventory item is the shifted inventory item based on the inventory item being shifted within the same location.

2. The method of claim 1 , further comprising, responsive to the determination that the inventory item is the shifted inventory item, determining that the inventory item does not need to be re-categorized.

3. The method of claim 1 , wherein the portion of the image includes a location of a cell in a map of the area of real space and the inventory item is located in the location of the cell in the map.

4. The method of claim 3 , wherein the trained location detection model determines that the inventory item has changed the position within the same location based on the inventory item changing the position while remaining within the location of the cell in the map.

5. The method of claim 1 ,
wherein the method further includes in response to the trained location detection model determining that another inventory item has not changed the position within the same location, but has rather changed a location, implementing a trained product category determination model to determine a product category for the inventory item, and
wherein the implementing of the trained product category determination model is performed prior to the implementing of the trained item classification model.

6. The method of claim 5 , wherein the implementing of the trained product category determination model includes:
inputting, to the trained product category determination model, the portion of the image of the area of real space (a) matching a location of a cell in which the inventory item is located in a map of the area of real space and (b) obtained in a current time interval; and
determining, by the trained product category determination model, the product category for the inventory item, which is located in the cell in the map of the area of real space from the portion of the image obtained in the current time interval.

7. The method of claim 1 , wherein the implementing of the trained location detection model includes:
inputting, to the trained location detection model, (i) a portion of a first image of the area of real space (a) matching a location of a cell in which the inventory item is located in a map of the area of real space and (b) obtained in a previous time interval and (ii) a portion of a second image of the area of real space (a) matching the location of the cell and (b) obtained in a current time interval; and
determining, by the trained location detection model, that the inventory item located in the cell has changed the position within the same location in dependence on the location of the inventory item in the first image and the second image.

8. The method of claim 1 , wherein the implementing of the trained item classification model includes:
inputting, to the trained item classification model, (i) the portion of the image of the area of real space (a) matching a location of a cell in which the inventory item is located in a map of the area of real space and (b) obtained in a current time interval, (ii) a determined product category of the inventory item, and (iii) a determined size of the inventory item; and
classifying, by the trained item classification model, the inventory item in dependence upon the portion of the image of the area of real space, the determined product category of the inventory item and the determined size of the inventory item.

9. The method of claim 8 , prior to the inputting to the trained item classification model, the method further including:
determining, using a trained item size detection model, a size of the inventory item, which is located in the cell, from the portion of the image obtained in the current time interval, by processing the portion of the image by warping and cropping the portion of the image and providing the processed portion of the image to the trained item classification model.

10. The method of claim 1 ,
wherein the received signal indicating the change is generated by matching at least one factored image of inventory locations in the area of real space captured in a first time interval with at least one factored image of inventory locations in the area of real space captured in a second time interval, and
wherein the factored images are generated by removing foreground objects occluding inventory items placed in inventory locations.

11. The method of claim 10 , wherein the received signal indicating the change is generated based on a mismatch being detected between the factored images at a location of the inventory item.

12. The method of claim 1 , wherein the received signal indicating the change is generated by matching a first image captured in a first time interval with a second image captured in a second time interval, such that the signal indicating the change is generated based on the first image including at least a hand, of a subject, without an inventory item and the second image containing the hand, of the subject, holding at least one inventory item.

13. The method of claim 1 , wherein the received signal indicating the change is generated by matching at least a first image of inventory locations in the area of real space captured in a first time interval with at least a second image of inventory locations in the area of real space captured in a second time interval, wherein the first image and the second image are captured by a same sensor, and wherein the received signal indicating the change is generated based on a mismatch being detected between the first image and the second image.

14. The method of claim 1 , wherein the trained location detection model determines that the inventory item has changed a position within the same location in the area of real space based on the inventory item being relocated to another location in the area of real space that is designated as being a proper location for a particular inventory item that is the same as the inventory item.

15. The method of claim 1 , wherein the inventory item data includes at least one of: a category of the inventory item, a sub-category of the inventory item, a description of the inventory item, a location in three-dimensions of the inventory item within the area of real space, a weight of the inventory item, a flavor of the inventory item, and a shelf-identifier of a shelf on which the inventory item is located.

16. A method for tracking inventory items in an area of real space, the method comprising:
in response to receiving a signal, generated in dependence on sensors and indicating a change to a portion of an image of an area of real space, implementing a trained location detection model to determine that an in

Assignees

• Standard Cognition Corp

Inventors

• Fisher Jordan
• Mazumder Nairwita
• Ghosh Arpan
• Singh Mohit Deep
• Woollard David
• Huang Michael

Classifications

• G06T7/20

  Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51) · CPC title

• G06T7/0002

  Inspection of images, e.g. flaw detection · CPC title

• G06Q10/0833

  Tracking · CPC title

• G06V20/70

  Labelling scene content, e.g. deriving syntactic or semantic representations · CPC title

• G06V10/242

  by image rotation, e.g. by 90 degrees · CPC title