Identifying user-item interactions in an automated facility

Patent metadata

Field	Value
Publication number	US-11087271-B1
Application number	US-201715470105-A
Country	US
Kind code	B1
Filing date	Mar 27, 2017
Priority date	Mar 27, 2017
Publication date	Aug 10, 2021
Grant date	Aug 10, 2021

Abstract

Techniques for employing user interfaces to output information indicative of events occurring in an inventory facility, and receive feedback from a human regarding the events are described herein. In one implementation, an event may take place in an inventory facility, such as a customer taking an item from an inventory location, returning an item to an inventory location, and so forth. An automated system of an inventory management system may process sensor data collected by sensors in the inventory facility to determine details of the event. In some examples, the inventory management system is unable to determine with a high level of confidence what occurred during the event. The inventory management system may provide the sensor data to a human associate through an associate interface, and receive input regarding details of the event from the human associate through the associate interface.

First claim

What is claimed is:

1. A method comprising:
receiving first video data at a client computing device, the first video data corresponding to a location of an environment for a period of time and depicting:
a first user at the location for at least a portion of the period of time;
a second user at the location for at least a portion of the period of time;
a first item at the location for at least a portion of the period of time; and
a second item at the location for at least a portion of the period of time;

sending data representing a user interface (UI) to a display associated with the client computing device, the UI comprising:
a first portion in which the first video data is presented; and
a second portion, adjacent the first portion, that includes a user-selection control, an item-selection control, and a submit-answer control;

receiving input indicating a selection of the user-selection control;
sending, at least partly in response to the receiving the input indicating the selection of the user-selection control, data to the display to be presented in the second portion of the UI, the data representing a first user image corresponding to the first user, a second user image corresponding to the second user, and first text corresponding to a request to identify whether the first user or the second user interacted with one of the first item or the second item depicted in the video data;
receiving input indicating a selection of the first user image;
storing an indication of the selection of the first user image;
receiving input indicating a selection of the item-selection control;
sending, at least partly in response to the receiving the input indicating the selection of the item-selection control, data to the display to be presented in the second portion of the UI, the data representing a first item image corresponding to the first item, a second item image corresponding to the second item, and second text corresponding to a request to identify whether the first user interacted with the first item or the second item;
receiving input indicating a selection of the first item image;
storing an indication of the selection of the first item image;
receiving input indicating a selection of the submit-answer control;
sending, to one or more server computing devices and at least partly in response to the receiving the input indicating the selection of the submit-answer control, data representing the indication of the selection of the first user image and the indication of the selection of the first item image.

2. The method of claim 1 , further comprising:
sending data representing a first item identifier to the display, the data representing the first item identifier configured to cause the first item identifier to display the first item identifier in the first portion of the UI at a position corresponding to the first item in the first video data;
sending data representing a second item identifier to the display, the data representing the second item identifier configured to cause the second item identifier to be presented in the first portion of the UI at a position corresponding to the second item in the first video data;
sending additional data representing the first item identifier to the display, the additional data representing the first item identifier configured to cause the first item identifier to be presented in the second portion of the UI adjacent to the first item image;
sending additional data representing the second item identifier to the display, the additional data representing the second item identifier configured to cause the second item identifier to be presented in the second portion of the UI adjacent to the second item image; and
sending data representing a bounding box to the display, the data representing the bounding box configured to cause the bounding box to be presented in the first portion of the UI, the bounding box surrounding at least the first item and the second item in the first video data.

3. The method of claim 1 , further comprising:
sending data representing a timeline to the display, the data representing the timeline configured to cause the timeline to be presented in the first portion of the UI, the timeline representing the period of time;
sending data representing an event bar to the display, the data representing the event bar configured to cause the event bar to be overlaid on at least a portion of the timeline, wherein a first end of the event bar is associated with a first point-in-time prior to the first user interacting with the first item and a second end of the event bar is associated with a second point-in-time after which the first user interacts with the first item; and
sending data representing an event indicator to the display, the data representing the event indicator configured to cause the event indicator to be presented adjacent the event bar, wherein a position of the event indicator corresponds to a third point-in-time, between the first and second points-in-time, at which the first user interacted with the first item.

4. The method of claim 1 , wherein the first video data was acquired by a first camera at a first position in the environment, and further comprising:
receiving second video data at the client computing device, the second video data corresponding at least partly to the location of the environment for at least a portion of the period of time, the second video data acquired by a second camera at a second position in the environment;
sending data including a representation of the second video data to the display, the data including the representation of the second video data configured to cause the representation of the second video data to be presented in a third portion of the UI adjacent the first portion;
receiving input indicating a selection of the representation of the second video data; and
sending the second video data to the display to be presented in the first portion of the UI.

5. A computing device comprising
one or more processors;
a display;
one or more computer-readable media storing instructions that, when executed by the one or more processors, cause the one or more processors to perform acts comprising:
receiving video data corresponding to a location of an environment for a period of time, the video data depicting a first user, a second user, and an item;
sending the video data to the display;
sending data including first text to the display, the first text corresponding to a request to identify whether the first user or the second user interacted with the item depicted in the video data; and
receiving a first input indicating that the first user interacted with the item depicted in the video data.

6. The computing device of claim 5 , the instructions further comprising instructions to cause the one or more processors to perform acts comprising:
receiving data representing a first user image corresponding to the first user;
receiving data representing a second user image corresponding to the second user;
sending the data representing the first user image to the display; and
sending the data representing the second user image to the display,
wherein receiving the first input indicating that the first user interacted with the item comprises receiving data indicating a selection of the first user image.

7. The computing device of claim 5 , the instructions further comprising instructions to cause the one or more processors to perform acts comprising:
sending, to the display, data representing a first instance of a first user identifier, the data representing the first instance of the first user identifier configured to cause the first instance of the first user identifier to be presented on the display at a location associated with a

Assignees

• Amazon Tech Inc

Inventors

• Guan Danny
• Mohanasundaram Prathiban
• Thomas Shannon
• Mandic Mirko
• Causse Helene
• Taylor Brett Richard
• Ahmed Waqas Syed
• Ignacio David Echevarria

Classifications

• G06F18/22

  Matching criteria, e.g. proximity measures · CPC title

• G06V40/172

  Classification, e.g. identification · CPC title

• G06V20/41

  Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49) · CPC title

• G06V20/44

  Event detection · CPC title

• G06Q10/087 Primary

  Inventory or stock management, e.g. order filling, procurement or balancing against orders · CPC title