Visual task feedback for workstations in materials handling facilities

Patent metadata

Field	Value
Publication number	US-9299013-B1
Application number	US-201414228127-A
Country	US
Kind code	B1
Filing date	Mar 27, 2014
Priority date	Mar 27, 2014
Publication date	Mar 29, 2016
Grant date	Mar 29, 2016

Abstract

Visual task feedback for workstations in a materials handling facility may be implemented. Image data of a workstation surface may be obtained from image sensors. The image data may be evaluated with regard to the performance of an item-handling task at the workstation. The evaluation of the image data may identify items located on the workstation surface, determine a current state of the item-handling task, or recognize an agent gesture at the workstation. Based, at least in part on the evaluation, one or more visual task cues may be selected to project onto the workstation surface. The projection of the selected visual task cues onto the workstation surface may then be directed.

First claim

What is claimed is:

1. A materials handling facility, comprising:
one or more imaging sensors, wherein the one or more imaging sensors include a sensor configured to capture geometric information of a workstation surface that corresponds to image data of the workstation surface;
one or more visual projectors;
a control system, configured to:
receive, from the one or more imaging sensors, image data of the workstation surface in the materials handling facility, wherein the geometric information corresponding to the image data is received along with the image data from the one or more imaging sensors;
evaluate the image data to identify one or more items located on the workstation surface associated with an item-handling task performed at the workstation, wherein to evaluate the image data to identify the one or more items located on the workstation surface, the control system is further configured to:
access a data store maintaining item description information regarding items handled in the materials handling facility; and
perform a comparative analysis based at least in part on the image data of the workstation surface of the image data and the item description information to identify the one or more items located on the workstation surface;

based, at least in part, on the one or more identified items, select one or more visual task cues to project onto the workstation surface, wherein the one or more visual task cues indicate a current state of the item-handling task associated with the one or more identified items; and
direct the one or more visual projectors to project the one or more visual task cues onto the workstation surface.

2. The materials handling facility of claim 1 , wherein the selected one or more visual task cues include at least one of:
an indication that an item associated with the item-handling task is missing from the workstation surface;
an indication that a particular one of the identified one or more items located on the workstation surface is incorrect for performance of the item handling task;
an indication that a particular one of the one or more items located on the workstation surface requires manual identification; or
an indication that the identified one or more items are complete for performance of the item handling task.

3. The system of claim 1 , wherein the materials handling facility includes a plurality of different workstations including the workstation, wherein different ones of the plurality of different workstations perform at least one different respective item-handling task, wherein the control system is configured to perform said receiving, said evaluating, said selecting, and said directing for at least some of the different ones of the plurality of different workstations in the materials handling facility.

4. A method, comprising:
performing, by one or more computing devices:
obtaining, via one or more imaging sensors, image data of a surface of a workstation in a materials handling facility;
evaluating the image data of the workstation surface with regard to a performance of an item-handling task performed at the workstation;
receiving location information for an agent performing the item-handling task at the workstation;
based, at least in part, on the location information, determining a viewing perspective for the agent;
based, at least in part, on the evaluation of the image data with regard to the performance of the item-handling task, determining one or more visual task cues to project onto the workstation surface; and
directing the projection of the one or more visual task cues onto the workstation surface, wherein said directing the projection of the one or more visual task cues onto the workstation surface is performed such that the one or more visual task cues are projected according to the viewing perspective of the agent.

5. The method of claim 4 , wherein said evaluation comprises performing at least one of:
identifying one or more items located on the workstation surface for the performance of the item-handling task;
determining a current state of the performance of the item-handling task; or
recognizing an agent gesture at the workstation associated with the performance of the item-handling task.

6. The method of claim 5 , wherein said identifying the one or more items located on the workstation surface associated with the item handling task performed at the workstation, comprises:
accessing a data store maintaining item description information regarding items handled in the materials handling facility; and
comparing the image data and the item description information to identify the one or more items located on the workstation surface.

7. The method of claim 6 , further comprising:
obtaining, via the one or more imaging sensors, geometric information of the workstation surface corresponding to the image data;
based, at least in part on the geometric information and the image data, calculating one or more item dimensions for the one or more items; and
wherein said comparing the image data and the item description information to identify the one or more items located on the workstation surface includes comparing the one or more item dimensions as part of said comparison.

8. The method of claim 5 , wherein said recognizing the agent gesture at the workstation associated with the performance of the item-handling task, comprises:
detecting an agent gesture with regard to a user interface projected on the workstation surface as an input action for the user interface.

9. The method of claim 4 , further comprising:
obtaining, via the one or more imaging sensors, additional image data of the workstation surface and geometric information of the workstation surface corresponding to the additional image data;
based, at least in part, on the additional image data of the workstation surface and the geometric information corresponding to the additional image data obtained from the one or more imaging sensors, detecting a surface change of the workstation surface that interferes with at least one of the projected one or more visual task cues;
determining one or more projection modifications for the at least one visual task cue; and
updating the at least one visual task cue projected onto the workstation according to the one or more projection modifications.

10. The method of claim 4 , wherein the one or more visual task cues are projected in an order indicating a sequence of actions for the performance of the item handling task.

11. The method of claim 4 , further comprising performing a respective one or more audio task cues in coordination with said projection of the determined one or more visual task cues.

12. The method of claim 4 , wherein the materials handling facility includes a plurality of different types of workstations each performing at least one different respective item-handling task, wherein the one or more computing devices perform said obtaining, said evaluating, said determining, and said directing for at least some of the plurality of different types of workstations in the materials handling facility.

13. A non-transitory, computer-readable storage medium, storing program instructions that when executed by one or more computing devices cause the one or more computing devices to at least implement:
obtaining, via one or more imaging sensors, image data of a surface of a workstation in a materials handling facility;
evaluating the image data of the workstation surface with regard to a performance of an item-handling task at the workstation based, at least in part on the image data of the workstation surface, wherein in said evaluating the image da

Assignees

• Amazon Tech Inc

Inventors

• Curlander James Christopher
• Colburn Robert Alexander

Classifications

• G06V40/28

  Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113) · CPC title

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06T7/50

  Depth or shape recovery · CPC title

• G06T7/60

  Analysis of geometric attributes · CPC title

• G06T7/74

  involving reference images or patches · CPC title