Self-checkout device that detects motion in video frames to register products present in the video

Patent metadata

Field	Value
Publication number	US-12380772-B2
Application number	US-202318357335-A
Country	US
Kind code	B2
Filing date	Jul 24, 2023
Priority date	Aug 3, 2022
Publication date	Aug 5, 2025
Grant date	Aug 5, 2025

Abstract

A self-checkout device and a method implemented by the self-checkout device is provided. The self-checkout device includes a detection plate to allow placement of product(s) thereon, and one or more cameras positioned to have a Field-of-View encompassing the detection plate to provide a video footage. A motion detection module detects presence of motion in the video footage. A sequence selection module selects a sequence of video frames over a time interval corresponding to the detection of the presence of motion in the video footage. An appearance interpretation module registers one or more products present in the sequence of video frames and a billing module fetches prices of the registered one or more products. Total bill is generated based on the fetched prices, and a payment is processed for the total bill.

First claim

The invention claimed is:

1. A self-checkout device comprising:
a detection plate adapted to allow placement of products thereon;
one or more cameras positioned to have a Field-of-View encompassing at least the detection plate, the one or more cameras configured to provide a video footage;
a processor; and
a memory communicably coupled to the processor and comprising processor instructions that when executed by the processor, wherein the processor comprises:
a motion detection module configured to detect presence of motion in the video footage;
a sequence selection module configured to select a sequence of video frames over a time interval corresponding to the detection of the presence of motion in the video footage;
an appearance interpretation module configured to register one or more products present in the sequence of video frames, wherein the appearance interpretation module comprises:
an object detection module configured to analyze the sequence of video frames to detect one or more objects therein;
a cropping module configured to isolate the detected one or more objects in the sequence of video frames and to extract visual features of the detected one or more objects;
an embedding module configured to convert the extracted visual features of the detected one or more objects into an embedded feature vector; and
an expert system module configured to compare the embedded feature vector with pre-stored feature vectors in an embedding database, and to identify the detected one or more objects based on the comparison,
wherein the identified one or more objects are registered as the one or more products;

a billing module configured to fetch prices of the registered one or more products, generate a total bill based on the fetched prices, and process a payment for the total bill; and
a controller module operatively connected to the one or more cameras and communicatively coupled with the motion detection module, the sequence selection module, the appearance interpretation module and the billing module to control operations thereof and facilitating communications therebetween.

2. The self-checkout device of claim 1 , wherein the appearance interpretation module employs a machine learning model to facilitate the detection, cropping, embedding, and identifying processes.

3. The self-checkout device of claim 1 , wherein the expert system module is further configured to determine if any one of the identified one or more objects is a weight-dependent loose product item from the one or more products.

4. The self-checkout device of claim 3 , further comprising a weighing module configured to activate a weighing scale unit to measure a weight of the weight-dependent loose product item from the one or more products placed on the detection plate, and wherein the billing module is configured to generate the total bill based on the measured weight of the weight-dependent loose product item.

5. The self-checkout device of claim 1 , further comprising a barcode processing module configured to detect one or more barcodes in the selected sequences of video frames and decode the detected barcodes corresponding to the registered one or more products, and wherein the billing module is configured to fetch prices of the registered one or more products based on the decoded barcodes.

6. The self-checkout device of claim 1 , further comprising a guidance module operatively connected to a design display unit, the guidance module configured to activate the design display unit to display a design on the detection plate to provide visual guidance to a user for optimal placement of products on the detection plate.

7. The self-checkout device of claim 1 , further comprising a concave mounting member disposed upright with respect to the detection plate, wherein the one or more cameras are mounted on the concave mounting member.

8. The self-checkout device of claim 7 , wherein the concave mounting member houses an illumination device to illuminate the detection plate.

9. The self-checkout device of claim 1 , wherein the one or more cameras comprises a first camera and a second camera oriented at different angles to capture the video footage of the products from multiple perspectives.

10. The self-checkout device of claim 1 , wherein the billing module is further configured to generate an itemized list based on the registered one or more products.

11. The self-checkout device of claim 10 , further comprising a display screen configured to display the itemized list and the total bill.

12. The self-checkout device of claim 1 , further comprising an admin module configured to support updates to configuration of the self-checkout device, including a product database thereof.

13. The self-checkout device of claim 1 , wherein the self-checkout device operates as a standalone device.

14. A method implemented by a self-checkout device, the method comprising:
receiving, by a processor, a video footage of a detection plate of the self-checkout device captured by one or more cameras, wherein the detection plate adapted to allow placement of products thereon, and
wherein the one or more cameras are positioned to have a Field-of-View encompassing at least the detection plate;
detecting, by the processor, a presence of motion in the video footage by processing thereof;
selecting, by the processor, a sequence of video frames over a time interval corresponding to the detection of the presence of motion in the video footage;
registering, by the processor, one or more products present in the sequence of video frames by:
analyzing, by the processor, the sequence of video frames to detect one or more objects therein;
isolating, by the processor, the detected one or more objects in the sequence of video frames and to extract visual features of the detected one or more objects;
converting, by the processor, the extracted visual features of the detected one or more objects into an embedded feature vector;
comparing, by the processor, the embedded feature vector with pre-stored feature vectors in an embedding database, and
identifying, by the processor, the detected one or more objects based on the comparison,
wherein the identified one or more objects are registered as the one or more products;

fetching, by the processor, prices of the registered one or more products;
generating, by the processor, a total bill based on the fetched prices and processing a payment for the total bill; and
displaying, by the processor, the total bill on a display screen of the self-checkout device.

15. The method according to claim 14 , further comprising:
detecting and decoding, by the processor, one or more barcodes visible in the sequence of video frames and in an event one or more barcodes are not visible therein, detecting, by the processor, from the one or more objects detected in the video frames, weight-dependent loose products and other items, wherein the other items comprise one of one or more sales items and one or more items that are not-for-sale including personal items belonging to a customer;
distinguishing between sales items and non-sales items of the detected items; and
issuing a first alert on detection of one or more non-sales items, the first alert comprising a message to remove the non-sales items placed on the detection plate of the self-checkout device.

16. The method according to claim 15 , further comprising:
determining, by the processor, a distribution of detected sales items on the detection plate of the self-checkout device; and
issuing, by the processor, a second alert on detecting that the determined d

Assignees

• Everseen Ltd

Inventors

• Parvu Ovidiu
• Cioarga Razvan-Dorel
• Hegarty Raymond

Classifications

• G06V20/68

  Food, e.g. fruit or vegetables · CPC title

• G06Q20/208 Primary

  Input by product or record sensing, e.g. weighing or scanner processing · CPC title

• G06Q20/201 Primary

  Price look-up processing, e.g. updating · CPC title

• G06V10/70

  using pattern recognition or machine learning (optical pattern recognition or electronic computations therefor G06V10/88) · CPC title

• G06V20/46

  Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames · CPC title