Self-service checkout counter checkout

Patent metadata

Field	Value
Publication number	US-11113680-B2
Application number	US-202016810670-A
Country	US
Kind code	B2
Filing date	Mar 5, 2020
Priority date	Apr 16, 2019
Publication date	Sep 7, 2021
Grant date	Sep 7, 2021

Abstract

Computer-implemented methods, non-transitory, computer-readable media, and computer-implemented systems for self-service checkout counter checkout. One computer-implemented method includes: obtaining, by using a camera, an image of at least one product placed on a checkout counter; performing image segmentation on the image to obtain at least one image region; identifying a product code included in a code region in an image region of the at least one image region; determining, based on the product code, a product category of a product associated with the product code; and determining a price of the product based on the product category.

First claim

What is claimed is:

1. A computer-implemented method for self-service checkout, comprising:
obtaining, by using a camera, an image, wherein the image is of at least one product placed on a checkout counter;
performing image segmentation on the image to obtain at least one image region;
identifying a product code included in a code region in an image region of the at least one image region;
determining that a product category of a product associated with the product code fails to be identified;
in response to determining that the product category fails to be identified, determining the product category by using an object detection model obtained by pre-training using labeled training sample images, each labeled training sample image comprising a rectangle that frames the product code to label the code region, wherein the object detection model generates a candidate image region in the image, performs target identification within the candidate image region, performs bounding box regression in the candidate image region, and corrects the code region by using a perspective transformation relative to position markers to obtain a corrected code region, and determines the product category based on the corrected code region; and
determining a price of the product based on the product category.

2. The computer-implemented method of claim 1 , wherein obtaining the image comprises:
controlling the camera to face the at least one product to obtain the image.

3. The computer-implemented method of claim 1 , wherein obtaining the image comprises receiving the image from a self-service checkout counter.

4. The computer-implemented method of claim 3 , wherein the image is taken from one of a top direction, a front direction, a left direction, a right direction, a rear direction, and an oblique direction of the at least one product, and an angle between the oblique direction and a direction vertical to a countertop of the self-service checkout counter is between 30 degrees to 60 degrees.

5. The computer-implemented method of claim 1 , wherein the image segmentation is performed by using an image segmentation model, and the image segmentation model is obtained by pre-training segmentation sample images comprising product images and annotations of silhouettes of products.

6. The computer-implemented method of claim 1 , wherein the product code is a barcode, and identifying the product code comprises:
detecting a barcode region in the image region by using an object detection model, the object detection model is obtained by pre-training training sample images comprising product images and annotations for identifying barcode regions in the product images.

7. The computer-implemented method of claim 6 , wherein identifying the product code comprises:
adjusting the barcode region based on the perspective transformation to obtain an adjusted barcode region; and
identifying the barcode in the adjusted barcode region.

8. The computer-implemented method of claim 1 , wherein the product code is a two-dimensional code, and identifying the product code comprises:
detecting position markers of the two-dimensional code in the image region; and
determining that a two-dimensional code region is detected in response to detecting at least two positioning markers.

9. The computer-implemented method of claim 8 , wherein identifying the product code comprises:
adjusting the two-dimensional code region based on the perspective transformation to obtain an adjusted two-dimensional code;
determining a corner-module relationship in the adjusted two-dimensional code based on the at least two positioning markers; and
identifying the two-dimensional code based on the corner-module relationship.

10. The computer-implemented method of claim 1 , wherein identifying the product category of the product associated with the product further comprises:
determining the product category based on an object detection model, wherein the object detection model is obtained by pre-training training sample images comprising product images and annotations for identifying products and product categories.

11. The computer-implemented method of claim 1 , wherein the image is a first image, the camera is a first camera, the at least one image region is a first at least one image region, the product is a first product, and the method further comprising:
obtaining, by using a second camera, a second image, wherein the second image is of the at least one product;
performing image segmentation on the second image to obtain a second at least one image region;
determining a product category of a second product associated with the second at least one image region based on a product code or visual identification; and
determining, based on a relative position between the first camera and the second camera, that the first at least one image region and the second at least one image region are associated with an identical product.

12. A computer-implemented system for self-service checkout, comprising:
one or more computers; and
one or more computer memory devices interoperably coupled with the one or more computers and having tangible, non-transitory, machine-readable media storing one or more instructions that, when executed by the one or more computers, perform one or more operations, comprising:
obtaining, by using a camera, an image, wherein the image is of at least one product placed on a checkout counter;
performing image segmentation on the image to obtain at least one image region;
identifying a product code included in a code region in an image region of the at least one image region;
determining that a product category of a product associated with the product code fails to be identified;
in response to determining that the product category fails to be identified, determining the product category by using an object detection model obtained by pre-training using labeled training sample images, each labeled training sample image comprising a rectangle that frames the product code to label the code region, wherein the object detection model generates a candidate image region in the image, performs target identification within the candidate image region, performs bounding box regression in the candidate image region, and corrects the code region by using a perspective transformation relative to position markers to obtain a corrected code region, and determines the product category based on the corrected code region; and
determining a price of the product based on the product category.

13. The computer-implemented system of claim 12 , wherein obtaining the image comprises:
controlling the camera to face the at least one product to obtain the image.

14. The computer-implemented system of claim 12 , wherein obtaining the image comprises receiving the image from a self-service checkout counter.

15. The computer-implemented system of claim 14 , wherein the image is taken from one of a top direction, a front direction, a left direction, a right direction, a rear direction, and an oblique direction of the at least one product, and an angle between the oblique direction and a direction vertical to a countertop of the self-service checkout counter is between 30 degrees to 60 degrees.

16. The computer-implemented system of claim 12 , wherein the image segmentation is performed by using an image segmentation model, the image segmentation model is obtained by pre-training segmentation sample images comprising product images and annotations of silhouettes of products.

17. The computer-implemented system of claim 12 , wherein t

Assignees

• Advanced New Technologies Co Ltd

Inventors

• Song Yang

Classifications

• G06K7/1404

  Methods for optical code recognition · CPC title

• G06T2207/20081

  Training; Learning · CPC title

• G06K7/1491

  the method including a reconstruction step, e.g. stitching two pieces of bar code together to derive the full bar code · CPC title

• G06Q20/208

  Input by product or record sensing, e.g. weighing or scanner processing · CPC title

• G06T7/12 Primary

  Edge-based segmentation · CPC title