Video analysis of food service counter operations

Patent metadata

Field	Value
Publication number	US-2020294018-A1
Application number	US-202016892238-A
Country	US
Kind code	A1
Filing date	Jun 3, 2020
Priority date	Dec 16, 2016
Publication date	Sep 17, 2020
Grant date	—

Abstract

In a restaurant having a customer-facing food service counter, a video camera is positioned to produce video of an area that includes the food service counter. The video is analyzed to determine the ingredients that are used to prepare each of multiple customer meals. This information may be used to recommend a modified ingredient layout of the food service counter, where the modified ingredient layout is formulated so as to improve serving efficiency or customer convenience.

First claim

1 . A method comprising:
obtaining, by one or more service computing devices associated with a merchant support service, inventory data associated with inventory of one or more inventory items associated with a merchant at a merchant location; receiving, by the one or more service computing and from a recording device, information associated with a change in inventory of an inventory item of the one or more inventory items; analyzing, by the one or more service computing devices, the information received from the recording device to identify an amount of depletion of the inventory item, wherein the depletion is via at least one of sale, transfer, consumption, or disposal; determining, by the one or more service computing devices, and based at least in part on the amount of depletion, an updated inventory of the inventory item at a second time; and generating, by the one or more service computing devices, one or more reports regarding the updated inventory of the inventory item.

2 . The method as claim 1 recites, wherein receiving the information from the recording device comprises receiving the information from the recording device by way of a point-of-sale (POS) device of the merchant.

3 . The method as claim 1 recites, wherein the one or more reports include a recommended action for the merchant based at least in part on the updated inventory.

4 . The method as claim 3 recites, wherein the recommended action comprises at least one of reordering, transferring, moving, purchasing, or manufacturing the inventory item.

5 . The method as claim 3 recites, wherein the inventory item comprises an item for sale, and wherein, based at least in part on the updated inventory of the inventory item being above a threshold amount, the recommended action comprises conducting a sales promotion for the inventory item.

6 . The method as claim 1 recites, wherein the inventory item is a food item in a food container, and wherein analyzing the information to identify an amount of depletion of the inventory item comprises determining a number of times the food container was accessed by an employee.

7 . The method as claim 1 recites, further comprising:
causing, by the one or more service computing devices, an interface of a merchant device to display the one or more reports.

8 . The method as claim 1 recites, further comprising:
determining, based at least in part on the inventory of the inventory item at the first time and the updated inventory of the item at the second time, a rate of depletion of the inventory item for the period between the first time and the second time;
estimating, based at least in part on the rate of depletion, an amount of time before the inventory of the inventory item will be below a threshold amount; and
sending a notification to the merchant of the amount of time

9 . One or more non-transitory computer-readable media storing instructions executable by one or more processors that, when executed by the one or more processors, cause the one or more processors to perform acts comprising:
obtaining, at a first time and by one or more service computing devices associated with a merchant support service and from a plurality of information capturing devices located in different locations within an establishment of a merchant, information associated with availability of an item, wherein the information comprises one or more of text data, video data, image data, or audio data; analyzing, by the one or more service computing devices, the information associated with the availability of the item, wherein analyzing the information comprises at least one of image recognition of the image data, image recognition of the video data, text recognition of the text data, or audio detection of the audio data; based at least in part on the availability of the item at the establishment at the first time, determining, at a second time by the one or more service computing devices, whether inventory data indicating inventory of the item at the establishment at the first time needs to be updated; and transmitting, by the one or more service computing devices to a merchant device associated with the merchant, a recommendation to update the inventory data.

10 . The one or more non-transitory computer-readable media as claim 9 recites, wherein the different locations within the establishment include one or more of a kitchen, a food service counter, a stock room, or a dining room.

11 . The one or more non-transitory computer-readable media as claim 9 recites, wherein one or more of the plurality of information capturing devices comprise one or more video recording devices, and wherein the data comprises one or more streams of data corresponding to the one or more video recording devices.

12 . The one or more non-transitory computer-readable media as claim 11 recites, wherein the one or more streams of data are received in real time or near real time.

13 . The one or more non-transitory computer-readable media as claim 9 recites, wherein the item is depleted by at least one of consumption, sale, transfer, or disposal.

14 . The one or more non-transitory computer-readable media as claim 13 recites, wherein the notification to the merchant of the amount of time comprises a recommendation to order more of the item to replenish a supply of the item before the inventory of the item is below the threshold amount, and wherein replenishing the supply of the item comprises at least one of reordering, transferring, moving, or manufacturing the item

15 . The one or more non-transitory computer-readable media as claim 14 recites, further comprising:
generating a reminder for the merchant to reorder the item on a particular date, wherein the particular date is based at least in part on the amount of time; and
sending the reminder to the merchant device on the particular date.

16 . The one or more non-transitory computer-readable media as claim 9 recites, wherein the item comprises a non-food item.

17 . A system for managing inventory in a restaurant environment comprising:
one or more processors; one or more non-transitory computer-readable media storing instructions executable by the one or more processors, wherein the instructions program the one or more processors to perform actions comprising:
obtaining, at a first time and by one or more service computing devices associated with a merchant support service and from a plurality of information capturing devices located in different locations within an establishment of a merchant, information associated with availability of an item, wherein the information comprises one or more of text data, video data, image data, or audio data;
analyzing, by the one or more service computing devices, the information associated with the availability of the item, wherein analyzing the information comprises at least one of image recognition of the image data, image recognition of the video data, text recognition of the text data, or audio detection of the audio data;
based at least in part on the availability of the item at the establishment at the first time, determining, at a second time by the one or more service computing devices, whether inventory data indicating inventory of the item at the establishment of the merchant at a first time needs to be updated; and
transmitting, by the one or more service computing devices to a merchant device associated with the merchant, a recommendation to update the inventory.

18 . The system as claim 17 recites, wherein one or more of the plurality of information captu

Assignees

• Square Inc

Inventors

• Wolter Jonathan Andrew

Classifications

• G06V40/172

  Classification, e.g. identification · CPC title

• G06V40/174

  Facial expression recognition · CPC title

• G06V20/40

  in video content (extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44) · CPC title

• G06V20/52

  Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69) · CPC title

• G06V20/68

  Food, e.g. fruit or vegetables · CPC title