System and method for detecting scan irregularities at self-checkout terminals

Patent metadata

Field	Value
Publication number	US-11100331-B2
Application number	US-201916254760-A
Country	US
Kind code	B2
Filing date	Jan 23, 2019
Priority date	Jan 23, 2019
Publication date	Aug 24, 2021
Grant date	Aug 24, 2021

Abstract

A system for detecting a scan irregularity in scanning process during check-out at a retail store, includes an image receiving module for receiving a video stream of a scanning zone, an image processing module for detecting visual scan intervals in image frames of the video stream, and a decision module. The decision module is configured to process each detected visual scan interval, wherein a processed visual scan interval includes a valid scan action, wherein the valid scan action is a user action performed for scanning an item. The decision module is further configured to detect a scan irregularity in the check-out process, wherein the scan irregularity occurs when an item identified for scanning in a processed visual scan interval is absent in a list of scanned items generated by the scanner during corresponding interval, and provide an alert regarding the scan irregularity at a user computing device.

First claim

The invention claimed is:

1. A system for detecting a scan irregularity in scanning of one or more items by a user using a scanner, the system comprising:
an image receiving module configured to receive a video stream of a scanning zone in real-time from at least one video camera, wherein the scanning zone is a region in a field of view of the scanner;
an image processing module configured to process each image frame of the video stream for detecting one or more visual scan intervals in one or more image frames, wherein the detected visual scan interval is a series of image frames in a time interval of the received video stream during which an item is identified, in the series of image frames, in the scanning zone scanned by the scanner; and
a decision module configured to:
process each detected visual scan interval based on a set of pre-defined rules to generate a processed visual scan interval which includes a scan action, and wherein the processed visual scan interval is generated based, at least, on a computed glass motion coverage of the one or more image frames of the visual scan interval;
detect a scan irregularity in the check-out process when an item identified, for scanning in the processed visual scan interval, is absent from a list of items generated by the scanner during the corresponding time interval; and
provide an alert regarding the scan irregularity at a user computing device.

2. The system of claim 1 , wherein the image processing module comprises:
a skin tone detector configured to detect a scan action in a current image frame based on presence of a human hand in the current image frame, wherein the presence of a human hand is detected based on a percentage of skin pixels in the current image frame relative to a previous image frame;
a motion detector configured to detect a scan action in the current image frame based on a movement of the human hand in the current image frame, wherein the motion is detected based on a percentage of motion pixels in the current image frame relative to the previous image frame; and
a key-point detector configured to detect a scan action in the current image frame based on presence of an object in the human hand in the current image frame, wherein the presence of the object is determined based on a number of key-points in the scanning zone, wherein a threshold on a temporal evolution of the number of key-points present in the scanning zone provides an estimate of a visual scan interval, and wherein the key-point detector detects a visual scan interval for the current image frame if a scan action is found in the current image frame.

3. The system of claim 1 , wherein the decision module is configured to set a pre-defined range of a visual scan interval based on a first pre-defined rule, wherein the decision module invalidates a detected visual scan interval that is outside the pre-defined range of the visual scan interval.

4. The system of claim 1 , wherein the decision module is configured to set a pre-defined threshold distance based on a second pre-defined rule, wherein the decision module merges two consecutive visual scan intervals if a distance between the two consecutive visual scan intervals is less than the pre-defined threshold distance.

5. The system of claim 1 , wherein the decision module is configured to take into account of a synchronization delay between the video camera and the scanner based on a third pre-defined rule, while processing a visual scan interval.

6. The system of claim 1 , wherein the the glass motion coverage is a ratio between a number of frames depicting a glass area of the scanning zone and the number of frames having the foreground other than the glass area.

7. The system of claim 1 , wherein the decision module is configured to validate a detected visual scan interval based on a percentage of skin pixels with respect to foreground pixels in corresponding image frames, based on a fifth pre-defined rule.

8. The system of claim 1 , wherein the decision module is configured to invalidate a detected visual scan interval if a number of key-points in corresponding image frames is less than a key-point threshold value, based on a sixth pre-defined rule.

9. A method for detecting a scan irregularity in scanning of one or more items by a user using a scanner, the method comprising:
receiving a video stream of a scanning zone in real-time from at least one video camera, wherein the scanning zone is a region in a field of view of the scanner;
processing each image frame of the video stream for detecting one or more visual scan intervals in one or more image frames, wherein the detected visual scan interval is a series of image frames in a time interval of the received video stream during which an item is identified, in the series of image frames, in the scanning zone scanned by the scanner;
processing each detected visual scan interval based on a set of pre-defined rules to generate a processed visual scan interval which includes a scan action, and wherein the processed visual scan interval is generated based, at least, on a computed glass motion coverage of the one or more image frames of the visual scan interval;
detecting a scan irregularity in the check-out process when an item identified, for scanning in the processed visual scan interval, is absent from a list of items generated by the scanner during the corresponding time interval; and
providing an alert regarding the scan irregularity at a user computing device.

10. The method of claim 9 , wherein the processing each image frame of the video stream comprises:
detecting a scan action in a current image frame based on presence of a human hand in the current image frame, wherein the presence of a human hand is detected based on a percentage of skin pixels in the current image frame relative to a previous image frame;
detecting a scan action in the current image frame based on movement of the human hand in the current image frame, wherein the motion is detected based on a percentage of motion pixels in the current image frame relative to the previous image frame; and
detecting a scan action in the current image frame based on presence of an object in the human hand in the current image frame, wherein the presence of the object is determined based on a number of key-points in the scanning zone, wherein a threshold on a temporal evolution of the number of key-points present in the scanning zone provides an estimate of a visual scan interval, and wherein the key-point detector detects a visual scan interval for the current image frame if a scan action is found in the current image frame.

11. The method of claim 9 further comprising setting a pre-defined range of scan interval based on a first pre-defined rule, wherein a detected visual scan interval is invalidated, that is outside the pre-defined range of scan interval.

12. The method of claim 9 further comprising setting a pre-defined threshold distance based on a second pre-defined rule for merging two consecutive visual scan intervals if a distance between the two consecutive visual scan intervals is less than the pre-defined threshold distance.

13. The method of claim 9 further comprising taking into account of a synchronization delay between the video camera and the scanner based on a third pre-defined rule, while processing a visual scan interval.

14. The method of claim 9 , wherein the computed glass motion coverage is a ratio between a number of frames depicting a glass area of the scanning zone and the number of frames having the foreground other than the glass area.

15. The method of claim 9 further comprising validatin

Assignees

• Everseen Ltd

Inventors

• Pricochi Andrei
• Ciprian David
• Cernazanu-Glavan Cosmin
• Pescaru Dan

Classifications

• G06V20/44

  Event detection · CPC title

• G06V20/46

  Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames · CPC title

• G06V40/107

  Static hand or arm · CPC title

• G06V20/52 Primary

  Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69) · CPC title

• G06V40/28

  Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113) · CPC title