Performing automated tasks based on visual cues

Patent metadata

Field	Value
Publication number	US-9846806-B2
Application number	US-201414315717-A
Country	US
Kind code	B2
Filing date	Jun 26, 2014
Priority date	Jun 26, 2014
Publication date	Dec 19, 2017
Grant date	Dec 19, 2017

Abstract

Visual cues may be provided for the purpose of triggering the performance of one or more actions associated with a given task. A visual cue may include one or more characters or other markings on any relevant surfaces, as well as naturally appearing edges, contours or outlines within an environment in which the visual cue is provided. When one or more visual cues appear within a field of view of an imaging device, and the visual cues are recognized as such, a task relating to the visual cues may be identified, and instructions for performing the task may be provided. The task may be dependent upon a context in which the visual cues are provided, such that a single visual cue may be associated with multiple tasks in different contexts, and the performance of the task may be confirmed based on imaging data or other relevant information.

First claim

What is claimed is:

1. An automated task performing system comprising:
an imaging device;
at least one data store;
an automated machine; and
a computing device in communication with the imaging device and the automated machine,
wherein the computing device has at least one computer processor and at least one memory having processor-executable instructions stored thereon, and
wherein the computing device, in response to execution of the processor-executable instructions by the at least one computer processor, is configured to at least:
capture at least a first image of an object using the imaging device;
recognize a first visual cue expressed within at least the first image, wherein the first visual cue is associated with at least a portion of the object;
determine a state or condition of the object based at least in part on the first visual cue;
determine an association between the state or condition of the object and at least one predetermined task to be performed upon the object based at least in part on information regarding a plurality of predetermined tasks maintained in the at least one data store;
identify at least one action required for performing the at least one predetermined task based at least in part on the association, wherein the at least one action is to be executed by the automated machine;
provide an electronic instruction for executing the at least one action to the automated machine;
identify a desired state or condition of an environment in which the object is provided, wherein the desired state or condition of the environment is associated with the at least one predetermined task;
capture at least a second image of the object using the imaging device;
determine a state or condition of the environment based at least in part on the second image;
determine that the state or condition of the environment is consistent with the desired state or condition of the environment; and
in response to determining that the state or condition of the environment is consistent with the desired state or condition of the environment,

store an indication that the at least one predetermined task has been performed.

2. The automated task performing system of claim 1 , wherein the computing device is further configured to at least:
identify a context associated with the first visual cue, wherein the association is determined based at least in part on the context.

3. The automated task performing system of claim 1 , wherein the first visual cue comprises at least one of:
an alphanumeric character;
a symbol; or
an icon,
and
wherein the first visual cue does not comprise a bar code.

4. A method comprising:
capturing a first set of imaging data regarding an object by at least one imaging device;
determining, by at least one computer processor, that the first set of imaging data comprises information regarding at least one visual cue associated with the object;
recognizing, by the at least one computer processor, the at least one visual cue in at least a portion of the first set of imaging data;
determining, by the at least one computer processor, a state or condition of the object based at least in part on the at least one visual cue;
identifying, by the at least one computer processor, a first automated task associated with the state or condition of the object based at least in part on the at least one visual cue;
providing, by the at least one computer processor, an instruction for performing at least one action associated with the first automated task to a first automated machine;
identifying a desired state or condition of an environment in which the object is provided, wherein the desired state or condition of the environment is associated with the first automated task;
capturing a second set of imaging data regarding the object by the at least one imaging device;
determining a state or condition of the environment based at least in part on the second set of imaging data;
determining whether the state or condition of the environment is consistent with the desired state or condition of the environment; and
in response to determining that the state or condition of the environment is consistent with the desired state or condition of the environment,
storing an indication that the first automated task has been performed.

5. The method of claim 4 , wherein the first automated machine comprises the at least one imaging device.

6. The method of claim 4 , wherein the first set of imaging data comprises at least one image of the object,
wherein the at least one visual cue is expressed within the at least one image of the object.

7. The method of claim 6 , further comprising:
presenting, by at least one human, at least one of the object or the at least one visual cue within a field of view of the at least one imaging device at a first time,
wherein the at least one image is captured by the at least one imaging device at the first time.

8. The method of claim 4 , wherein recognizing the at least one visual cue in at least the portion of the first set of imaging data comprises:
recognizing at least one of an edge, a contour or an outline of the object based at least in part on the first set of imaging data; and
identifying the at least one visual cue based at least in part on the edge, the contour or the outline.

9. The method of claim 4 , wherein identifying the first automated task associated with the state or condition of the object comprises:
identifying a plurality of records stored in at least one data store, wherein each of the records comprises an association between one of a plurality of visual cues and at least one of a plurality of automated tasks;
selecting at least one of the plurality of records corresponding to the state or condition of the object; and
identifying the first automated task based at least in part on the selected at least one of the plurality of records.

10. The method of claim 4 , wherein identifying the first automated task associated with the object comprises:
determining a context associated with at least one of the at least one visual cue, the object or the environment,
wherein the first automated task is identified based at least in part on the context.

11. A computer-implemented method comprising:
capturing a first set of imaging data regarding an object by at least one imaging device;
determining, by at least one computer processor, that the first set of imaging data comprises information regarding at least one visual cue associated with the object;
identifying, by the at least one computer processor, the at least one visual cue based at least in part on the first set of imaging data;
identifying, by the at least one computer processor, a first automated task associated with the object based at least in part on the at least one visual cue;
providing, by the at least one computer processor, an instruction for performing at least one action associated with the first automated task to a first automated machine;
identifying a desired condition of an environment in which the object is provided, wherein the desired condition of the environment is associated with the first automated task;
capturing a second set of imaging data regarding the object by the at least one imaging device;
determining a condition of the environment in which the object is provided based at least in part on the second set of imaging data;
determining whether the condition of the environment is consistent with the desired condition of the environment; and
upon determining that the condition of the environment is consistent with the desired condition of the environment,
storing an indication that the first automated task

Assignees

• Amazon Tech Inc

Inventors

• Mishra Pragyana K
• Graybill Jules Cook
• Guan Danny

Classifications

• G06Q10/06 Primary

  Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling · CPC title

• G06V20/80

  Recognising image objects characterised by unique random patterns · CPC title

• G06K2209/19

  Physics · mapped topic

• G06K9/00624

  Physics · mapped topic

• G06K9/00577 Primary

  Physics · mapped topic