Merging events in interactive data processing systems

Patent metadata

Field	Value
Publication number	US-11568361-B1
Application number	US-202117350741-A
Country	US
Kind code	B1
Filing date	Jun 17, 2021
Priority date	Sep 27, 2016
Publication date	Jan 31, 2023
Grant date	Jan 31, 2023

Abstract

This disclosure describes interactive data processing systems configured to facilitate selection by a human associate of tentative results generated by an automated system from sensor data. In one implementation, an event may take place in a materials handling facility. The event may comprise a pick or place of an item from an inventory location, movement of a user, and so forth. The sensor data associated with the event is processed by an automated system to determine tentative results associated with the event. In some situations, an uncertainty may exist as to which of the tentative results accurately reflects the actual event. The system may then determine whether the event is to be merged with one or more temporally and spatially proximate events and, if so, the sensor data and tentative results for the merged event is sent to a human associate. The associate may select one of the tentative results.

First claim

What is claimed is:

1. A computing device comprising:
a display;
one or more processors; and
one or more computer-readable media storing computer-executable instructions that, when executed on the one or more processors, cause the one or more processors to perform acts comprising:
presenting, on the display, first data representing a first event occurring in an environment;
presenting, on the display, second data representing a second event occurring in the environment, wherein the first event and second event occurred within a threshold period of time;
presenting, on the display, a first tentative result of a merged event that includes the first event and the second event;
presenting, on the display, a second tentative result of the merged event that is different than the first tentative final result;
presenting, on the display, inquiry data including a request for information indicative of a final result of the merged event; and
receiving input indicating the final result of the first event and the second event.

2. The computing device of claim 1 , the acts further comprising:
receiving weight data indicating a change in weight on a weight sensor in the environment, the change in weight is associated with an item being moved relative to the weight sensor,
wherein presenting the first tentative result includes presenting an item identifier associated with the item.

3. The computing device of claim 1 , the acts further comprising:
receiving the first data that includes image data depicting a user moving at least one of a first item or a second item with respect to a location within the environment,
wherein presenting the first tentative result includes presenting an item identifier associated with the first item.

4. The computing device of claim 1 , the acts further comprising:
receiving the first data that includes image data depicting a user moving at least one of a first item or a second item with respect to a location within the environment,
wherein presenting the first tentative result includes presenting an item identifier associated with the first item.

5. The computing device of claim 1 , the acts further comprising:
receiving the first data that includes image data depicting a first user and a second user at a location in the environment at which an item was moved,
wherein presenting the first tentative result includes presenting a user identifier associated with the first user indicating that the first user moved the item.

6. The computing device of claim 1 , the acts further comprising:
receiving the first tentative result associated with the first event, the first tentative result indicating a first item was interacted with by a user; and
receiving the second tentative result associated with the second event, the second tentative result indicating a second item was interacted with by the user,
wherein:
presenting the first tentative result includes presenting a first item identifier associated with the first item; and
presenting the second tentative result includes presenting a second item identifier associated with the second item.

7. The computing device of claim 1 , the acts further comprising:
receiving the first tentative result associated with the first event, the first tentative result indicating a first user interacted with an item; and
receiving the second tentative result associated with the second event, the second tentative result indicating a second user interacted with the item,
wherein:
presenting the first tentative result includes presenting a first user identifier associated with the first user; and
presenting the second tentative result includes presenting a second user identifier associated with the second user.

8. A method comprising:
receiving first sensor data representing a first event occurring in an environment, the first event being associated with a first time range;
receiving second sensor data representing a second event occurring in the environment, the second event being associated with a second time range that is within a threshold amount of time of the first time range;
presenting, on a display associated with a client computing device, first image data representing the first event;
presenting, on the display, second image data representing the second event;
presenting, on the display, inquiry data including a request for information indicative of a final result of at least the first event and the second event; and
receiving, at the client computing device, input indicating the final result for the first event and the second event.

9. The method of claim 8 , wherein the first sensor data is weight data indicating a change in weight on a weight sensor in the environment, further comprising:
receiving a tentative result indicating that the change in weight is associated with an item being moved relative to the weight sensor; and
presenting an item identifier associated with the item on the display.

10. The method of claim 8 , wherein the first sensor data is video data depicting a location within the environment, and further comprising:
receiving a tentative result indicating that the video data depicts an item being moved with respect to the location; and
presenting an item identifier associated with the item on the display.

11. The method of claim 8 , further comprising:
receiving a first tentative result associated with the first event, the first tentative result indicating a first item was interacted with by a user; and
receiving a second tentative result associated with the second event, the second tentative result indicating a second item was interacted with by the user, wherein:
presenting, on the display, a first item identifier associated with the first item; and
presenting, on the display, a second item identifier associated with the second item.

12. The method of claim 8 , further comprising:
receiving a first tentative result associated with the first event, the first tentative result indicating a first user interacted with an item; and
receiving a second tentative result associated with the second event, the second tentative result indicating a second user interacted with the item, wherein:
presenting, on the display, a first user identifier associated with the first user; and
presenting, on the display, a second user identifier associated with the second user.

13. The method of claim 8 , wherein the receiving the first sensor data comprises receiving video data depicting a user interacting with an item within the environment, further comprising:
presenting graphical overlay on the display that depicting at least one of the item or the user.

14. A system comprising:
a display;
one or more processors; and
one or more computer-readable media storing computer-executable instructions that, when executed on the one or more processors, cause the one or more processors to perform acts comprising:
presenting, on the display, first data representing a first event occurring in an environment;
presenting, on the display, second data representing a second event occurring in the environment, wherein the first event and second event occurred within a threshold period of time;
presenting, on the display, a first tentative result of a merged event that includes the first event and the second event;
presenting, on the display, a second tentative result of the merged event that is different than the first tentative final result;
presenting, on the display, inquiry data including a request for information indicative of a final result of the merged event; and
receiving input indicating the final result of the fir

Assignees

• Amazon Tech Inc

Inventors

• Xiong Fei
• Iyer Sabarishkumar
• Ignacio David Echevarria
• Boyapati Sridhar
• Lee Caylan
• Thomas Jacques Daniel
• Ahmed Waqas Syed
• Guan Danny

Classifications

• G06F16/5854

  using shape and object relationship · CPC title

• G06V30/194

  References adjustable by an adaptive method, e.g. learning · CPC title

• G06K9/6201

  Physics · mapped topic

• H04N7/18

  Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast · CPC title

• G06K9/6288

  Physics · mapped topic