Teaching a robot to place objects in dynamic environments

Patent metadata

Field	Value
Publication number	US-9977965-B1
Application number	US-201715633700-A
Country	US
Kind code	B1
Filing date	Jun 26, 2017
Priority date	Jun 26, 2017
Publication date	May 22, 2018
Grant date	May 22, 2018

Abstract

A method may include obtaining first images, each image including an object, and determining a set of one or more visual cues for each. The method may include selecting a common visual cue of the one or more visual cues that is common to each set of one or more visual cues determined for each corresponding image and determining a correlation between a location of the common visual cue in each image of the first images and a location of the object in each image of the first images. The method may include obtaining a second image of an environment and identifying the common visual cue in the second image. The method may include determining a placement location for the object in the environment based on the correlation and a location of the common visual cue in the second image.

First claim

What is claimed is:

1. A method of placing an object in a location, the method comprising:
obtaining a first plurality of images, each image of the first plurality of images including an object;
determining a set of one or more visual cues for each image of the first plurality of images;
selecting a common visual cue of the one or more visual cues that is common to each set of one or more visual cues determined for each corresponding image;
determining a correlation between a location of the common visual cue in each image of the first plurality of images and a location of the object in each image of the first plurality of images;
obtaining a second image of an environment;
identifying the common visual cue in the second image; and
based on the correlation and based on a location of the common visual cue in the second image, determining a placement location for the object in the environment.

2. The method of claim 1 , further comprising instructing a robotic device to place the object at the placement location in the environment.

3. The method of claim 2 , further comprising the robotic device placing the object at the placement location in the environment.

4. The method of claim 1 , wherein the environment is different in at least one respect from each environment represented by each image of the first plurality of images.

5. The method of claim 1 , wherein the common visual cue is in one location of at least one image of the first plurality of images and is in a different location of at least one other image of the first plurality of images.

6. The method of claim 1 , further comprising obtaining demonstration data from observation of a human, wherein determining the placement location for the object in the environment is further based on the demonstration data.

7. The method of claim 1 , further comprising obtaining one or more audio cues from a human, wherein determining the placement location for the object in the environment is further based on the one or more audio cues.

8. The method of claim 1 , further comprising obtaining one or more directional cues from a human, wherein determining the placement location for the object in the environment is further based on the one or more directional cues.

9. A non-transitory computer-readable medium having computer-executable instructions stored thereon that are executable by a processor to perform or control performance of operations comprising:
obtaining a first plurality of images, each image of the first plurality of images including an object;
determining a set of one or more visual cues for each image of the first plurality of images;
selecting a common visual cue of the one or more visual cues that is common to each set of one or more visual cues determined for each corresponding image;
determining a correlation between a location of the common visual cue in each image of the first plurality of images and a location of the object in each image of the first plurality of images;
obtaining a second image of an environment;
identifying the common visual cue in the second image; and
based on the correlation and based on a location of the common visual cue in the second image, determining a placement location for the object in the environment.

10. The non-transitory computer-readable medium of claim 9 , wherein the operations further comprise: instructing a robotic device to place the object at the placement location in an environment represented by the second image.

11. The non-transitory computer-readable medium of claim 10 , wherein the operations further comprise: the robotic device placing the object at the placement location in the environment.

12. The non-transitory computer-readable medium of claim 9 , wherein the environment is different in at least one respect from each environment represented by each image of the first plurality of images.

13. The non-transitory computer-readable medium of claim 9 , wherein the common visual cue is in one location of at least one image of the first plurality of images and is in a different location of at least one other image of the first plurality of images.

14. The non-transitory computer-readable medium of claim 9 , wherein the operations further comprise: obtaining demonstration data from observation of a human, wherein determining the placement location for the object in the environment is further based on the demonstration data.

15. The non-transitory computer-readable medium of claim 9 , wherein the operations further comprise: obtaining one or more audio cues from a human, wherein determining the placement location for the object in the environment is further based on the one or more audio cues.

16. The non-transitory computer-readable medium of claim 9 , wherein the operations further comprise: obtaining one or more directional cues from a human, wherein determining the placement location for the object in the environment is further based on the one or more directional cues.

17. A system to place an object in a location, the system comprising:
a camera;
a learning module, the learning module configured to:
obtain a first plurality of images, each image of the first plurality of images including an object;
determine a set of one or more visual cues for each image of the first plurality of images;
select a common visual cue of the one or more visual cues that is common to each set of one or more visual cues determined for each corresponding image;
determine a correlation between a location of the common visual cue in each image of the first plurality of images and a location of the object in each image of the first plurality of images;

a replay module communicatively coupled to the camera and the learning module, the replay module configured to:
obtain a second image of an environment from the camera;
identify the common visual cue in the second image; and
based on the correlation and based on a location of the common visual cue in the second image, determine a placement location for the object in the second image; and

a robotic device communicatively coupled to the replay module and configured to place the object at the placement location in the environment in accordance with instructions received from the replay module.

18. The system of claim 17 , the system further comprising a demonstration module communicatively coupled to the learning module and to the robotic device, the demonstration module configured to:
obtain demonstration data from observation of a human for placement of the object from the robotic device;
determine a placed location for the object from the demonstration data; and
provide the placed location to the learning module;
and wherein determining the placement location for the object in the environment is further based on the placed location.

19. The system of claim 17 , the system further comprising a microphone communicatively coupled to the learning module, the microphone configured to obtain one or more audio cues from a human, and wherein determining the placement location for the object in the environment is further based on the one or more audio cues.

20. The system of claim 17 , wherein the learning module is further configured to obtain one or more directional cues from a human from the camera, and wherein determining the placement location for the object in the environment is further based on the one or more directional cues.

Assignees

• Fujitsu Ltd

Inventors

• Parikh Samir

Classifications

• G06V10/776 Primary

  Validation; Performance evaluation · CPC title

• G06V20/30

  in albums, collections or shared content, e.g. social network photos or video · CPC title

• G06F18/217

  Validation; Performance evaluation; Active pattern learning techniques · CPC title

• G05B2219/40564

  Recognize shape, contour of object, extract position and orientation · CPC title

• G05B2219/40154

  Moving of objects · CPC title