Self-checkout system, method thereof and device therefor

Patent metadata

Field	Value
Publication number	US-2019371134-A1
Application number	US-201916425961-A
Country	US
Kind code	A1
Filing date	May 30, 2019
Priority date	Jun 1, 2018
Publication date	Dec 5, 2019
Grant date	—

Abstract

A self-checkout system capable of product identification and customer abnormal behavior detection, a method thereof and a device therefor are provided herein. The self-checkout system includes a product identification device and a customer abnormal behavior detection device. The product identification device is configured to perform a product identification, in which whether products are correctly placed on a platform and whether the identification can be completed are determined. The customer abnormal behavior detection device is configured to detect whether a customer has an abnormal checkout behavior.

First claim

1 . A self-checkout system, comprising:
a platform, configured to place at least one product; a product identification device, configured to perform a product identification on the at least one product placed on the platform; and a customer abnormal behavior detection device, configured to perform an abnormal checkout behavior detection based on a customer image captured in front of the platform to obtain an abnormal behavior detection result, wherein when determining that the abnormal behavior detection result is an abnormal behavior, an abnormal behavior notification is sent to thereby adjust the abnormal behavior.

2 . The self-checkout system according to claim 1 , wherein the customer abnormal behavior detection device comprises:
at least one image capturing unit, configured to capture the customer image; and a processor, configured to perform the abnormal checkout behavior detection on the customer image to obtain the abnormal behavior detection result, wherein the abnormal checkout behavior detection comprises performing a posture identification process to detect a checkout posture in the customer image, and then performing a handheld object identification process on a region based on the checkout posture to obtain the abnormal behavior detection result.

3 . The self-checkout system according to claim 2 , wherein before performing the posture identification process, the processor of the customer abnormal behavior detection device performs a real-time keypoint detection process on the customer image to obtain keypoint information of a customer in the customer image for performing the posture identification process.

4 . The self-checkout system according to claim 3 , wherein the processor is configured to obtain a body keypoint line of the customer from the customer image, and comparing the body keypoint line with a preset model to obtain the keypoint information.

5 . The self-checkout system according to claim 2 , wherein the processor of the customer abnormal behavior detection device is configured to obtain a plurality of key points in the customer image, and compare a key point line formed by the key points with a preset model to obtain the checkout posture corresponding to a customer.

6 . The self-checkout system according to claim 5 , wherein the processor of the customer abnormal behavior detection device further obtains a human body posture category based on the checkout posture, and determines a position and a range of a handheld object candidate region for performing the handheld object identification process.

7 . The self-checkout system according to claim 1 , wherein the product identification device performs the product identification on the at least one product placed on the platform to obtain an identification result, wherein if the identification result is not obtained, a prompt notification is sent for adjusting a placement manner of the at least one product on the platform.

8 . The self-checkout system according to claim 1 , wherein the product identification device is configured to start to perform the product identification by identifying a customer gesture in the customer image through a camera, or is configured to start to perform the product identification by determining whether a customer is close to the platform through an infrared ray sensing, an ultrasonic wave sensing or a microwave sensing.

9 . The self-checkout system according to claim 1 , wherein the product identification device is configured to project a serial number onto the at least one product.

10 . The self-checkout system according to claim 7 , wherein the product identification device comprises:
an image capturing unit, capturing a platform image of the at least one product placed on the platform; and a processor, performing the product identification on the platform image to obtain a plurality of features corresponding to the at least one product, and performing a comparison with a product feature database based on the features to obtain the identification result.

11 . The self-checkout system according to claim 10 , wherein when the processor of the product identification device performs the product identification on the platform image to obtain the features corresponding to the at least one product for performing the comparison to obtain the identification result, if a number of the features is insufficient, the prompt notification is sent for adjusting the placement manner of the at least one product on the platform.

12 . The self-checkout system according to claim 11 , wherein the processor of the product identification device is configured to segment a plurality of product regions in the platform image by an edge detection, detect the features of the at least one product from the product regions, and identify the features of the at least one product.

13 . The self-checkout system according to claim 12 , wherein when performing the product identification on the platform image, the processor of the product identification device is configured to obtain a classification result confidence value by comparing the platform image with the product feature database, and obtain the identification result if the classification result confidence value is greater than a threshold.

14 . A self-checkout method, comprising:
performing a product identification on at least one product placed on a platform; capturing a customer image; and performing an abnormal checkout behavior detection based on the customer image, and obtaining an abnormal behavior detection result based on the customer image, wherein when determining that the abnormal behavior detection result is an abnormal behavior, an abnormal behavior notification is sent to thereby adjust the abnormal behavior.

15 . The self-checkout method according to claim 14 , wherein the abnormal checkout behavior detection comprises performing a posture identification process to detect a checkout posture in the customer image, and then performing a handheld object identification process on a region based on the checkout posture to obtain the abnormal behavior detection result.

16 . The self-checkout method according to claim 15 , wherein before the posture identification process, a real-time keypoint detection process is performed on the customer image to obtain keypoint information of a customer in the customer image for performing the posture identification process.

17 . The self-checkout method according to claim 16 , wherein the real-time keypoint detection process obtains a body keypoint line of the customer from the customer image, and compares the body keypoint line with a preset model to obtain the keypoint information.

18 . The self-checkout method according to claim 15 , wherein the handheld object identification process comprises obtaining a plurality of key points in the customer image, and comparing a key point line formed by the key points with a preset model to obtain the checkout posture corresponding to a customer.

19 . The self-checkout method according to claim 18 , wherein a position and a range of a handheld object candidate region are further determined based on the checkout posture for performing the handheld object identification process.

20 . The self-checkout method according to claim 14 , further comprising capturing a platform image of the at least one product on the platform, obtaining an identification result based on the platform image, and sending a prompt notification for adjusting a placement manner of the at least one product when t

Assignees

• Ind Tech Res Inst

Inventors

• Chen Ming-Yen
• Lin Chang-Hong
• Yang Hsin-Yeh
• Hsiao Po-Hsuan

Classifications

• G07G1/0036 Primary

  Checkout procedures · CPC title

• G06Q20/18

  involving self-service terminals [SST], vending machines, kiosks or multimedia terminals · CPC title

• G06Q20/208

  Input by product or record sensing, e.g. weighing or scanner processing · CPC title

• G06Q20/405

  Establishing or using transaction specific rules · CPC title

• G06Q20/3278 Primary

  RFID or NFC payments by means of M-devices · CPC title