Video analysis of food service counter operations

Patent metadata

Field	Value
Publication number	US-11334858-B2
Application number	US-202016892238-A
Country	US
Kind code	B2
Filing date	Jun 3, 2020
Priority date	Dec 16, 2016
Publication date	May 17, 2022
Grant date	May 17, 2022

Abstract

In a restaurant having a customer-facing food service counter, a video camera is positioned to produce video of an area that includes the food service counter. The video is analyzed to determine the ingredients that are used to prepare each of multiple customer meals. This information may be used to recommend a modified ingredient layout of the food service counter, where the modified ingredient layout is formulated so as to improve serving efficiency or customer convenience.

First claim

The invention claimed is:

1. A method comprising:
obtaining, by one or more service computing devices associated with a merchant support service, inventory data associated with inventory of one or more inventory items associated with a merchant at a merchant location;
receiving, by the one or more service computing and from an information capturing device, information associated with a change in inventory of an inventory item of the one or more inventory items by one or more persons associated with the merchant at the merchant location, wherein the information comprises at least one of image data, video data, text data, or audio data;
analyzing, by the one or more service computing devices, the information received from the information capturing device to quantify a change in an amount of the inventory item, wherein the change occurs when a designated location of the inventory item at the merchant location is accessed by the one or more persons in association with at least one of sale, purchase, transfer, consumption, or disposal, and wherein analyzing the information comprises at least one of image recognition, text recognition, or audio detection;
determining, by the one or more service computing devices, and based at least in part on the change, an updated inventory of the inventory item at a second time; and
generating, by the one or more service computing devices, one or more reports regarding the updated inventory of the inventory item.

2. The method as claim 1 recites, wherein receiving the information from the information capturing device comprises receiving the information from the information capturing device by way of a point-of-sale (POS) device of the merchant.

3. The method as claim 1 recites, wherein the one or more reports include a recommended action for the merchant based at least in part on the updated inventory, and wherein generating the one or more reports is responsive at least in part to determining that the updated inventory of the inventory item is beyond a threshold amount.

4. The method as claim 3 recites, wherein the recommended action comprises at least one of reordering, transferring, moving, purchasing, selling, or manufacturing the inventory item.

5. The method as claim 3 recites, wherein determining the updated inventory of the inventory item is beyond the threshold amount comprises determining the updated inventory is above a threshold amount, and wherein the recommended action comprises conducting a sales promotion for the inventory item.

6. The method as claim 1 recites, wherein the inventory item is a food item and the designated location is a food container, and wherein analyzing the information to quantify the change in the amount of the inventory item comprises determining a number of times the food container was accessed by the one or more persons.

7. The method as claim 1 recites, further comprising:
causing, by the one or more service computing devices, an interface of a merchant device to display the one or more reports.

8. The method as claim 1 recites, further comprising:
determining, based at least in part on the inventory of the inventory item at a first time and the updated inventory of the inventory item at the second time, a rate of depletion of the inventory item for a period between the first time and the second time;
estimating, based at least in part on the rate of depletion, an amount of time before the inventory of the inventory item will be below a threshold amount; and
sending a notification to the merchant of the amount of time.

9. One or more non-transitory computer-readable media storing instructions executable by one or more processors that, when executed by the one or more processors, cause the one or more processors to perform acts comprising:
obtaining, at a first time and by one or more service computing devices associated with a merchant support service and from a plurality of information capturing devices located in different locations within an establishment of a merchant, information associated with availability of an item when a location of the item within the establishment is accessed by one or more persons associated with the merchant at the establishment, wherein the information comprises one or more of text data, video data, image data, or audio data;
analyzing, by the one or more service computing devices, the information associated with the availability of the item when the location of the item within the establishment is accessed by the one or more persons, wherein analyzing the information comprises at least one of image recognition of the image data, image recognition of the video data, text recognition of the text data, or audio detection of the audio data;
based at least in part on the availability of the item at the establishment at the first time, determining, at a second time by the one or more service computing devices, whether inventory data indicating inventory of the item at the establishment at the first time needs to be updated; and
transmitting, by the one or more service computing devices to a merchant device associated with the merchant, a recommendation to update the inventory data.

10. The one or more non-transitory computer-readable media as claim 9 recites, wherein the different locations within the establishment include one or more of a kitchen, a food service counter, a stock room, or a dining room.

11. The one or more non-transitory computer-readable media as claim 9 recites, wherein one or more of the plurality of information capturing devices comprise one or more video recording devices, and wherein the information comprises one or more streams of data corresponding to the one or more video recording devices.

12. The one or more non-transitory computer-readable media as claim 11 recites, wherein the one or more streams of data are received in real time or near real time.

13. The one or more non-transitory computer-readable media as claim 9 recites, further comprising:
determining, based at least in part on analyzing the information associated with the availability of the item, a decreased availability of the item, wherein the decreased availability caused by the one or more persons accessing the location of the item in association with at least one of consumption, sale, transfer, or disposal.

14. The one or more non-transitory computer-readable media as claim 13 recites, wherein the recommendation comprises a recommendation to order more of the item to replenish a supply of the item before the inventory of the item is below a threshold amount, and wherein replenishing the supply of the item comprises at least one of reordering, transferring, moving, or manufacturing the item.

15. The one or more non-transitory computer-readable media as claim 14 recites, further comprising:
generating a reminder for the merchant to reorder the item on a particular date, wherein the particular date is based at least in part on an amount of time before the inventory of the item is below the threshold amount; and
sending the reminder to the merchant device on the particular date.

16. The one or more non-transitory computer-readable media as claim 9 recites, wherein the item comprises a non-food item.

17. A system for managing inventory in a restaurant environment comprising:
one or more processors;
one or more non-transitory computer-readable media storing instructions executable by the one or more processors, wherein the instructions program the one or more processors to perform actions comprising:
obtaining, at a first time and by one or more service computing device

Assignees

• Block Inc
• Square Inc

Inventors

• Wolter Jonathan Andrew

Classifications

• G07G1/0054

  with control of supplementary check-parameters, e.g. weight or number of articles · CPC title

• G06Q10/0875

  Itemisation or classification of parts, supplies or services, e.g. bill of materials · CPC title

• G06V40/28

  Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113) · CPC title

• G06V20/40

  in video content (extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44) · CPC title

• G06V40/174

  Facial expression recognition · CPC title