System and method for self-checkout using product images

Patent metadata

Field	Value
Publication number	US-10129507-B2
Application number	US-201414331565-A
Country	US
Kind code	B2
Filing date	Jul 15, 2014
Priority date	Jul 15, 2014
Publication date	Nov 13, 2018
Grant date	Nov 13, 2018

Abstract

A network of cameras is disposed throughout a store, such as a grocery store, for example. The cameras are positioned so as to monitor the products available in the store, as well as the entrances and exits to and from the store. Customers are identified as they enter the store, and their movements throughout the store are tracked. The cameras capture images of the products selected by the customer and then send those images to a computer server. The server identifies each product selected by the customer, maintains a total price for all products selected by the customer, and automatically debits the customer's account for the total price upon detecting the customer leaving the store.

First claim

What is claimed is:

1. A method for checking out products from a store, the method implemented by an image processing server and comprising:
receiving a plurality of images of an product from one or more cameras focused on products positioned on a predetermined section of a shelf in a store responsive to a customer removing the product from the predetermined section of the shelf and placing the product into a cart, wherein each of the one or more cameras is communicatively connected to a corresponding image processing server, and wherein each of the plurality of images of the product is captured from a different angle by different cameras focused on the predetermined section of the shelf;
generating a multi-dimensional image from the plurality of images of the product being removed from the predetermined section of the shelf by the customer;
identifying the product removed by the customer based on an image analysis of the generated multi-dimensional image;
recording the identified product removed by the customer as a purchase made by the customer based on the image analysis;
determining a predetermined confidence level threshold value based on one or both of a customer activity and a known shopping history for the customer, the predetermined confidence value defining a minimum level of confidence in which products placed in the customer's cart must be accurately identified;
determining a confidence level value representing a current level of confidence that the product placed in the customer's cart has been accurately identified;
determining whether the current confidence level value for having accurately identified the product in the customer's cart meets or exceeds the predetermined confidence level threshold value;
responsive to determining that the current confidence level value does not meet or exceed the predetermined confidence level threshold value:
analyzing information printed on a label of the product to identify the product; and
triggering an audit of the customer responsive to determining, based on the analysis, that the product has not been accurately identified to within the predetermined confidence level; and

automatically debiting an account associated with the user for a price of the product responsive to determining that the current confidence level value meets or exceeds the predetermined confidence level threshold value.

2. The method of claim 1 wherein recording the identified product removed by the customer as a purchase made by the customer comprises:
determining the price for the product removed by the customer;
adding the price for the identified product removed by the customer to a total price for all products selected by the customer; and
sending one or both of the price and the total price to a device associated with the customer.

3. The method of claim 2 further comprising deducting the price of the identified product removed by the customer from the total price responsive to detecting the customer returning the product to the predetermined section of the shelf.

4. The method of claim 2 further comprising triggering an audit of the products removed by the customer responsive to detecting the customer returning the product to an area of the store that is different than the predetermined section of the shelf.

5. The method of claim 1 wherein identifying the product removed by the customer based on an image analysis of the multi-dimensional image comprises identifying the product based on a result of comparing the multi-dimensional image to one or more images of the product stored in a memory circuit.

6. An image processing server for a store, the image processing server having one or more cameras assigned thereto and comprising:
a communications interface configured to communicate with one or more cameras disposed throughout the store, wherein each of the one or more cameras is focused on a respective predetermined section of a shelf in the store; and
a processor circuit configured to:
receive a plurality of images of a product from the one or more cameras focused on products positioned on a first predetermined section of a shelf in the store responsive to detecting that a customer removed the product from the first predetermined section of a shelf in the store and placed the product into a cart, wherein each of the plurality of images of the product is captured from a different angle by different cameras focused on the predetermined section of the shelf;
generate a multi-dimensional image from the plurality of images of the product being removed from the first predetermined section of the shelf by the customer;
identify the product removed from the first predetermined section of the shelf in the store by the customer based on an image analysis of the generated multi-dimensional image;
record the identified product from the first predetermined section of the shelf as a purchase made by the customer based on the image analysis
determine a predetermined confidence level threshold value based on one or both of a customer activity and a known shopping history for the customer, the predetermined confidence value defining a minimum level of confidence in which products placed in the customer's cart must be accurately recorded;
determine a confidence level value representing a current level of confidence that the product placed in the customer's cart has been accurately identified;
determine whether the current confidence level value for having accurately identified the product in the customer's cart meets or exceeds the predetermined confidence level threshold value;
responsive to determining that the current confidence level value does not meet or exceed the predetermined confidence level threshold value:
analyze information printed on a label of the product to identify the product; and
trigger an audit of the customer responsive to determining, based on the analysis, that the product has not been accurately identified to within the predetermined confidence level; and

automatically debiting an account associated with the user for a price of the product responsive to determining that the current confidence level value meets or exceeds the predetermined confidence level threshold value.

7. The computer server of claim 6 wherein the processor circuit is further configured to:
determine the price for the product removed by the customer;
add the price for the identified product to a total price for all products removed by the customer; and
sending one or both of the price and the total price to a device associated with the customer.

8. The computer server of claim 7 wherein the processor circuit is further configured to deduct the price of the identified product removed by the customer from the total price responsive to detecting the customer returning the product to the first predetermined section of a shelf.

9. The computer server of claim 7 wherein the processor circuit is further configured to trigger an audit of the customer responsive to detecting the customer returning the product to an area of the store that is different than the first predetermined section of a shelf.

10. The computer server of claim 6 further comprising a memory circuit configured to store one or more images of the product, and wherein the processor circuit is further configured to identify the product based on a comparison of the generated multi-dimensional image to one or more images of the product stored in the memory circuit.

11. A method for checking out products from a store having a plurality of cameras and a plurality of image processing servers, the method comprising:
assigning each camera to at least one image processing server, wherein each camera is

Assignees

• Toshiba Global Commerce Solutions Holdings Corp

Inventors

• Landers Jr John David
• Steiner David John
• Do Phuc

Classifications

• G06Q20/201 Primary

  Price look-up processing, e.g. updating · CPC title

• G06Q20/202

  Interconnection or interaction of plural electronic cash registers [ECR] or to host computer, e.g. network details, transfer of information from host to ECR or from ECR to ECR · CPC title

• G06K9/00771

  Physics · mapped topic

• H04N7/181 Primary

  for receiving images from a plurality of remote sources · CPC title

• G07G1/0045

  with a code reader for reading of an identifying code of the article to be registered, e.g. barcode reader or radio-frequency identity [RFID] reader · CPC title