Self-serve beverage detection and assignment

Patent metadata

Field	Value
Publication number	US-11893757-B2
Application number	US-202017104925-A
Country	US
Kind code	B2
Filing date	Nov 25, 2020
Priority date	Oct 25, 2019
Publication date	Feb 6, 2024
Grant date	Feb 6, 2024

Abstract

An image sensor is positioned such that a field-of-view of the sensor encompasses portion of a beverage machine. The field-of-view includes a zone associated with operating the beverage machine. A subsystem receives a signal indicating beverage is dispensed and angled-view images from the image sensor. First image(s) associated with a start of beverage being dispensed are determined by detecting a hand of a person entering the zone associated with operating the beverage machine. A first identifier is determined for the person in the first image(s). Second image(s) associated with an end of the beverage being dispensed are determined by detecting a hand of the person exiting the zone. A second identifier is determined for the person in the second image(s). If the first identifier is the same as the second identifier, the beverage is assigned to the person with the first identifier.

First claim

The invention claimed is:

1. A system, comprising:
a beverage machine operable to dispense a beverage and communicate a signal indicating a beverage is dispensed;
an image sensor positioned such that a field-of-view of the image sensor encompasses at least a portion of the beverage machine, wherein the image sensor is configured to generate angled-view images of at least a portion of the beverage machine, wherein the portion is associated with operating the beverage machine to dispense the beverage; and
a beverage assignment subsystem coupled to the image sensor and the beverage machine, the beverage assignment subsystem comprising at least one processor configured to:
receive the signal communicated by the beverage machine indicating the beverage is dispensed;
determine a first one or more image frames associated with a start of the beverage being dispensed by detecting a hand of a person entering a zone associated with the portion of the beverage machine captured in the angled-view images;
determine, based on the one or more image frames associated with the start of the beverage being dispensed, a first identifier of the person whose hand entered the zone associated with the portion of the beverage machine captured in the angled-view images;
determine a second one or more image frames associated with an end of the beverage being dispensed by detecting a hand of a person exiting the zone associated with the portion of the beverage machine captured in the angled-view images;
determine, based on the second one or more image frames, a second identifier of the person whose hand exited the zone associated with the portion of the beverage machine captured in the angled-view images;
if the first identifier is the same as the second identifier, assign the beverage to the person associated with the first identifier; and
if the first identifier is not the same as the second identifier, do not assign the beverage to the person associated with the first identifier.

2. The system of claim 1 , wherein the processor is further configured to, if the first identifier is the same as the second identifier, assign the beverage to the person by adding a beverage item to a digital shopping cart associated with the first identifier.

3. The system of claim 1 , wherein the processor is further configured to, prior to assigning the beverage to the person if the first identifier is the same as the second identifier, determine, that a time interval between the start of the beverage being dispensed and the end of the beverage being dispensed is at least a threshold value.

4. The system of claim 1 , wherein the processor is further configured to, prior to assigning the beverage to the person if the first identifier is the same as the second identifier:
determine, based on properties of the person in a third one or more image frames following removal of a cup from the zone associated with the portion of the beverage machine captured in the angled-view images, a third identifier of the person whose hand removed the cup from the zone associated with the portion of the beverage machine captured in the angled-view images; and
determine that the third identifier is the same as the first and second identifiers.

5. The system of claim 1 , wherein the processor is further configured to determine the first one or more image frames associated with the start of the beverage being dispensed by:
detecting, based on the angled-view images generated by the image sensor, that the person is within a threshold distance of the beverage machine;
determining, based on the angled-view images generated by the image sensor, a hand position of the person using a pose estimation algorithm; and
determining that the hand position enters the zone associated with the portion of the beverage machine captured in the angled-view images.

6. The system of claim 1 , wherein the processor is further configured to determine the second one or more image frames associated with the end of the beverage being dispensed by determining that the hand position exits the zone associated with the portion of the beverage machine captured in the angled-view images.

7. The system of claim 1 , wherein:
the beverage machine comprises at least one visible marker positioned and configured to identify the zone associated with the portion of the beverage machine captured in the angled-view images; and
the processor is further configured to:
detect the at least one marker in an angled-view image received from the image sensor; and
automatically determine, based on the detected marker, an extent of one or both of the first zone and the second zone.

8. The system of claim 1 , wherein the zone associated with the portion of the beverage machine captured in the angled-view images comprises a dispensing mechanism of the beverage machine.

9. A method, comprising:
receiving a signal communicated by a beverage machine indicating the beverage is dispensed, wherein the beverage machine is operable to dispense a beverage and communicate a signal indicating a beverage is dispensed;
determining a first one or more image frames from an image sensor associated with a start of the beverage being dispensed by detecting a hand of a person entering a zone associated with the portion of the beverage machine captured in the angled-view images, wherein the image sensor is positioned such that a field-of-view of the image sensor encompasses at least a portion of the beverage machine, wherein the image sensor is configured to generate angled-view images of at least a portion of the beverage machine, wherein the portion is associated with operating the beverage machine to dispense the beverage;
determining, based on the one or more image frames associated with the start of the beverage being dispensed, a first identifier of the person whose hand entered the zone associated with the portion of the beverage machine captured in the angled-view images;
determining a second one or more image frames associated with an end of the beverage being dispensed by detecting a hand of a person exiting the zone associated with the portion of the beverage machine captured in the angled-view images;
determining, based on the second one or more image frames, a second identifier of the person whose hand exited the zone associated with the portion of the beverage machine captured in the angled-view images;
if the first identifier is the same as the second identifier, assigning the beverage to the person associated with the first identifier; and
if the first identifier is not the same as the second identifier, not assigning the beverage to the person associated with the first identifier.

10. The method of claim 9 , further comprising, if the first identifier is the same as the second identifier, assigning the beverage to the person by adding a beverage item to a digital shopping cart associated with the first identifier.

11. The method of claim 9 , further comprising, prior to assigning the beverage to the person if the first identifier is the same as the second identifier, determining, that a time interval between the start of the beverage being dispensed and the end of the beverage being dispensed is at least a threshold value.

12. The method of claim 9 , further comprising, prior to assigning the beverage to the person if the first identifier is the same as the second identifier:
determining, based on properties of the person in a third one or more image frames following removal of a cup from the zone associated with the portion of the beverage machine captured in the angled-view images, a third identifier of the person whose hand removed the cup from the zone associated with the

Assignees

• 7 Eleven Inc

Inventors

• Wang Xinan
• Mirza Shahmeer Ali
• Krishnamurthy Sailesh Bharathwaaj

Classifications

• G06T7/70 Primary

  Determining position or orientation of objects or cameras (camera calibration G06T7/80) · CPC title

• G06T7/292

  Multi-camera tracking · CPC title

• G06V10/225

  based on a marking or identifier characterising the area · CPC title

• G06V20/41

  Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49) · CPC title

• G06V20/52

  Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69) · CPC title