System and method for detecting scan irregularities at self-checkout terminals

Patent metadata

Field	Value
Publication number	US-11854265-B2
Application number	US-202117376931-A
Country	US
Kind code	B2
Filing date	Jul 15, 2021
Priority date	Jan 23, 2019
Publication date	Dec 26, 2023
Grant date	Dec 26, 2023

Abstract

A system for detecting a scan irregularity in scanning process during check-out at a retail store, includes an image receiving module for receiving a video stream of a scanning zone, an image processing module for detecting visual scan intervals in image frames of the video stream, and a decision module. The decision module is configured to process each detected visual scan interval, wherein a processed visual scan interval includes a valid scan action, wherein the valid scan action is a user action performed for scanning an item. The decision module is further configured to detect a scan irregularity in the check-out process, wherein the scan irregularity occurs when an item identified for scanning in a processed visual scan interval is absent in a list of scanned items generated by the scanner during corresponding interval, and provide an alert regarding the scan irregularity at a user computing device.

First claim

The invention claimed is:

1. A retail store with a self-check-out process for one or more items, the retail store comprising:
a self-checkout terminal comprising:
a scanner configured for use in operably scanning an item when presented within a field of view of the scanner; and
at least one video camera configured to capture a video stream of a scanning zone in real-time, wherein the scanning zone is representative of the field of view of the scanner; and

a system communicably coupled to the scanner and the at least one video camera, the system configured to detect a scan irregularity in the scanning of the one or more items by a user using the scanner, the system comprising:
an image receiving module configured to receive the video stream of the scanning zone in real-time from the at least one video camera;
an image processing module configured to process each image frame of the video stream for detecting one or more visual scan intervals in one or more image frames, wherein each detected visual scan interval is a series of image frames in a time interval of the received video stream during which an item is identified, in the series of image frames, in the scanning zone scanned by the scanner; and
a decision module configured to:
process each detected visual scan interval based on a set of pre-defined rules to generate a processed visual scan interval which includes a scan action and wherein the processed visual scan interval is generated based, at least, on a computed glass motion coverage of the one or more image frames of the visual scan interval;
detect a scan irregularity in the self-check-out process when an item identified, for scanning in the processed visual scan interval, is absent from a list of items generated by the scanner during the corresponding time interval; and
provide an alert regarding the scan irregularity on at least one of: a user computing device and a display of the self-checkout terminal.

2. The retail store of claim 1 , wherein the image processing module comprises:
a skin tone detector configured to detect a scan action in a current image frame based on a presence of a human hand in the current image frame, wherein the presence of a human hand is detected based on a percentage of skin pixels in the current image frame relative to a previous image frame;
a motion detector configured to detect a scan action in the current image frame based on a movement of the human hand in the current image frame, wherein the movement is detected based on a percentage of motion pixels in the current image frame relative to the previous image frame; and
a key-point detector configured to detect a scan action in the current image frame based on a presence of an object in the human hand in the current image frame, wherein the presence of the object is determined based on a number of key-points in the scanning zone, wherein a threshold on a temporal evolution of the number of key-points present in the scanning zone provides an estimate of a visual scan interval, and wherein the key-point detector detects a visual scan interval for the current image frame if a scan action is found in the current image frame.

3. The retail store of claim 1 , wherein the decision module is configured to set a pre-defined range of a visual scan interval based on a first pre-defined rule, wherein the decision module invalidates a detected visual scan interval that is outside the pre-defined range of the visual scan interval.

4. The retail store of claim 1 , wherein the decision module is configured to set a pre-defined threshold distance based on a second pre-defined rule, wherein the decision module merges two consecutive visual scan intervals if a distance between the two consecutive visual scan intervals is less than the pre-defined threshold distance.

5. The retail store of claim 1 , wherein the decision module is configured to take into account of a synchronization delay between the video camera and the scanner based on a third pre-defined rule, while processing a visual scan interval.

6. The retail store of claim 1 , wherein the computed glass motion coverage is a ratio between a number of frames depicting a glass area of the scanning zone and the number of frames having a foreground other than the glass area.

7. The retail store of claim 1 , wherein the decision module is configured to validate a detected visual scan interval based on a percentage of skin pixels with respect to foreground pixels in corresponding image frames, based on a fifth pre-defined rule.

8. The retail store of claim 1 , wherein the decision module is configured to invalidate a detected visual scan interval if a number of key-points in corresponding image frames is less than a key-point threshold value, based on a sixth pre-defined rule.

9. The retail store of claim 1 , wherein the alert regarding the detected scan irregularity is provided in at least one of: an instant message, an e-mail, or a short messaging service (SMS) prompt.

10. A method of operating a retail store, the method comprising:
providing a self-checkout terminal having a scanner, at least one video camera, and a system coupled in communication with the scanner and the at least one video camera for detecting a scan irregularity in a self-check-out process;
scanning an item when presented within a field of view of the scanner;
capturing a video stream of a scanning zone in real-time using the at least one video camera, wherein the scanning zone is representative of at least the field of view of the scanner; and
receiving the video stream of the scanning zone in real-time from the at least one video camera by the system;
processing each image frame of the video stream for detecting one or more visual scan intervals in one or more image frames, wherein each detected visual scan interval is a series of image frames in a time interval of the received video stream during which an item is identified, in the series of image frames, in the scanning zone scanned by the scanner;
processing each detected visual scan interval based on a set of pre-defined rules to generate a processed visual scan interval which includes a scan action and wherein the processed visual scan interval is generated based, at least, on a computed glass motion coverage of the one or more image frames of the visual scan interval;
detecting a scan irregularity in the self-check-out process when an item identified, for scanning in the processed visual scan interval, is absent from a list of items generated by the scanner during the corresponding time interval; and
providing an alert regarding the scan irregularity on at least one of: a user computing device and a display of the self-checkout terminal.

11. The method of claim 10 , wherein the processing each image frame of the video stream comprises:
detecting a scan action in a current image frame based on a presence of a human hand in the current image frame, wherein the presence of a human hand is detected based on a percentage of skin pixels in the current image frame relative to a previous image frame;
detecting a scan action in the current image frame based on a movement of the human hand in the current image frame, wherein the movement is detected based on a percentage of motion pixels in the current image frame relative to the previous image frame; and
detecting a scan action in the current image frame based on a presence of an object in the human hand in the current image frame, wherein the presence of the object is determined based on a number of key-points in the scanning zone, wherein a threshold on a temporal evolution of the number of key-points present in the scanning zone provides an estimate of a visual scan interval, an

Assignees

• Everseen Ltd

Inventors

• Pricochi Andrei
• David Ciprian
• Cernazanu-Glavan Cosmin
• Pescaru Dan Alexandru

Classifications

• G06V20/52 Primary

  Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69) · CPC title

• G06V20/46

  Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames · CPC title

• G06V40/107

  Static hand or arm · CPC title

• G06V40/28

  Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113) · CPC title

• G06V20/44

  Event detection · CPC title