Space extrapolation for robot task performance

Patent metadata

Field	Value
Publication number	US-10058997-B1
Application number	US-201615184725-A
Country	US
Kind code	B1
Filing date	Jun 16, 2016
Priority date	Jun 16, 2016
Publication date	Aug 28, 2018
Grant date	Aug 28, 2018

Abstract

Methods, apparatus, systems, and computer-readable media are provided for enabling users to approximately identify a space within an environment inhabited by a plurality of objects that user wishes for a robot to manipulate. In various implementations, an approximation of a space within an environment may be identified based on user input. The actual space within the environment may then be extrapolated based at least in part on the approximation and one or more attributes of the environment. A plurality of objects that are co-present within the space and that are to be manipulated by a robot may be identified. The robot may then be operated to manipulate the identified plurality of objects.

First claim

What is claimed is:

1. A computer-implemented method, comprising:
identifying, by one or more processors, an approximation of a space within an environment, wherein identifying the approximation comprises:
detecting, based on visual data sensed by one or more vision sensors, an operator gesture associated with user identification of the space within the environment, and
temporally synchronizing the operator gesture with audio data sensed by one or more audio sensors,
wherein the operator gesture is detected in the visual data based at least in part by identifying one or more frames of the visual data that correspond temporally with one or more utterances captured in the audio data that are associated with user identification of the space;

extrapolating, by one or more of the processors, the space within the environment based on the approximation and one or more attributes of the environment;
identifying, by one or more of the processors, a plurality of objects that are co-present within the space, wherein the plurality of objects are to be manipulated by a robot; and
operating, by one or more of the processors, the robot to manipulate the identified plurality of objects.

2. The computer-implemented method of claim 1 , further comprising performing, by one or more of the processors, semantic processing of the audio data to associate the one or more utterances with user identification of the space.

3. The computer-implemented method of claim 1 , further comprising identifying, by one or more of the processors, based on the approximation, at least one surface in the environment that is associated with the space.

4. The computer-implemented method of claim 3 , further comprising:
detecting, by one or more of the processors, one or more edges associated with the at least one surface; and
defining, by one or more of the processors, one or more boundaries of the space based at least in part on the one or more detected edges.

5. The computer-implemented method of claim 4 , wherein detecting the one or more edges comprises performing edge detection on visual data sensed by one or more vision sensors.

6. The computer-implemented method of claim 4 , wherein detecting the one or more edges is based on one or more surfaces identified within a three-dimensional model of the environment available to the one or more processors.

7. The computer-implemented method of claim 1 , wherein the operator gesture comprises a pointed finger, and the approximation comprises a point at the end of a vector projected along the pointed finger until it intersects a distinct surface of the environment, wherein an entirety of the distinct surface is extrapolated from the point.

8. The computer-implemented method of claim 1 , wherein the at least one surface is a horizontal surface.

9. The computer-implemented method of claim 1 , wherein the at least one surface is a vertical surface to which the plurality of objects are secured or mounted.

10. The computer-implemented method of claim 1 , wherein the plurality of objects are heterogeneous.

11. A system comprising one or more processors and memory operably coupled with the one or more processors, wherein the memory stores instructions that, in response to execution of the instructions by the one or more processors, cause the one or more processors to:
identify, based on user input, an approximation of a space within an environment, wherein identification of the approximation comprises:
detection, based on visual data sensed by one or more vision sensors, of an operator gesture associated with user identification of the space within the environment, and
temporally synchronization of the operator gesture with audio data sensed by one or more audio sensors,
wherein the operator gesture is detected in the visual data based at least in part by identifying one or more frames of the visual data that correspond temporally with one or more utterances captured in the audio data that are associated with user identification of the space;

extrapolate, based on one or more attributes of the environment, the space within the environment based at least in part on the approximation;
identify a plurality of objects that are co-present within the space, wherein the plurality of objects are to be manipulated by a robot; and
operate the robot to manipulate the identified plurality of objects.

12. The system of claim 11 , further comprising instructions to identify, based on the approximation, at least one surface in the environment that is associated with the space.

13. At least one non-transitory computer-readable medium comprising instructions that, in response to execution of the instructions by a computing system, cause the computing system to perform the following operations:
identifying, based on user input, an approximation of a space within an environment, wherein identifying the approximation comprises:
detecting, based on visual data sensed by one or more vision sensors, an operator gesture associated with user identification of the space within the environment, and
temporally synchronizing the operator gesture with audio data sensed by one or more audio sensors,

wherein the operator gesture is detected in the visual data based at least in part by identifying one or more frames of the visual data that correspond temporally with one or more utterances captured in the audio data that are associated with user identification of the space;
extrapolating the space within the environment based at least in part on one or more attributes of the environment and the approximation;
identifying a plurality of objects that are co-present within the space, wherein the plurality of objects are to be manipulated by a robot; and
operating the robot to manipulate the identified plurality of objects.

Assignees

• X Dev Llc

Inventors

• Chao crystal

Classifications

• G05B2219/45063

  Pick and place manipulator · CPC title

• G06F3/017

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title

• B25J9/1679

  characterised by the tasks executed · CPC title

• G05B2219/39466

  Hand, gripper, end effector of manipulator · CPC title

• B25J9/1661 Primary

  characterised by task planning, object-oriented languages · CPC title