Virtual teach and repeat mobile manipulation system

Patent metadata

Field	Value
Publication number	US-11580724-B2
Application number	US-201916570852-A
Country	US
Kind code	B2
Filing date	Sep 13, 2019
Priority date	Jul 23, 2019
Publication date	Feb 14, 2023
Grant date	Feb 14, 2023

Abstract

A method for controlling a robotic device is presented. The method includes positioning the robotic device within a task environment. The method also includes mapping descriptors of a task image of a scene in the task environment to a teaching image of a teaching environment. The method further includes defining a relative transform between the task image and the teaching image based on the mapping. Furthermore, the method includes updating parameters of a set of parameterized behaviors based on the relative transform to perform a task corresponding to the teaching image.

First claim

What is claimed is:

1. A method of controlling a robotic device, comprising:
positioning the robotic device within a task environment;
mapping a plurality of task image pixel descriptors associated with a first group of pixels in a task image of a scene in the task environment to a plurality of teaching image pixel descriptors associated with a second group of pixels in a teaching image of a teaching environment, each task image pixel descriptor of the plurality of task image pixel descriptors comprising a first pixel value associated with a pixel in the first group of pixels, and each teaching image pixel descriptor of the plurality of teaching image pixel descriptors comprising a second pixel value associated with a pixel in the second group of pixels, the first pixel value associated with each task image pixel descriptor having a same value as the second pixel value associated with the teaching image pixel descriptor mapped to the respective task image pixel descriptor;
defining a relative transform between the task image and the teaching image based on mapping the plurality of task image pixel descriptors, the relative transform indicating a change in an x-axis, y-axis, z-axis, roll, pitch, and yaw between the task image and the teaching image; and
updating parameters of a set of parameterized behaviors based on the relative transform to perform a task corresponding to the teaching image.

2. The method of claim 1 , in which the set of parameterized behaviors comprises behaviors performed by a user while the robotic device was trained to perform the task with a virtual reality interface.

3. The method of claim 1 , in which the plurality of task image pixel descriptors are mapped to the plurality of teaching image pixel descriptors at an interval.

4. The method of claim 1 , in which:
a first position of the robotic device in the task environment is different from a second position of the robotic device during training in the teaching environment.

5. The method of claim 4 , in which the first position and the second position are associated with one or both of a starting location or pose of the robotic device.

6. The method of claim 4 , in which the first position and the second position are associated with one or both of a starting location or pose of an object on which the task is performed.

7. The method of claim 1 , in which the task environment and the teaching environment are associated with a same environment.

8. An apparatus for controlling a robotic device, the apparatus comprising:
a memory; and
at least one processor coupled to the memory, the at least one processor configured:
to position the robotic device within a task environment;
to map a plurality of task image pixel descriptors associated with a first group of pixels in a task image of a scene in the task environment to a plurality of teaching image pixel descriptors associated with a second group of pixels in a teaching image of a teaching environment, each task image pixel descriptor of the plurality of task image pixel descriptors comprising a first pixel value associated with a pixel in the first group of pixels, and each teaching image pixel descriptor of the plurality of teaching image pixel descriptors comprising a second pixel value associated with a pixel in the second group of pixels, the first pixel value associated with each task image pixel descriptor having a same value as the second pixel value associated with the teaching image pixel descriptor mapped to the respective task image pixel descriptor;
to define a relative transform between the task image and the teaching image based on mapping the plurality of task image pixel descriptors, the relative transform indicating a change in an x-axis, y-axis, z-axis, roll, pitch, and yaw between the task image and the teaching image; and
to update parameters of a set of parameterized behaviors based on the relative transform to perform a task corresponding to the teaching image.

9. The apparatus of claim 8 , in which the set of parameterized behaviors comprises behaviors performed by a user while the robotic device was trained to perform the task with a virtual reality interface.

10. The apparatus of claim 8 , in which the plurality of task image pixel descriptors are mapped to the plurality of teaching image pixel descriptors at an interval.

11. The apparatus of claim 8 , in which:
a first position of the robotic device in the task environment is different from a second position of the robotic device during training in the teaching environment.

12. The apparatus of claim 11 , in which the first position and the second position are associated with one or both of a starting location or pose of the robotic device.

13. The apparatus of claim 11 , in which the first position and the second position are associated with one or both of a starting location or pose of an object on which the task is performed.

14. The apparatus of claim 8 , in which the task environment and the teaching environment are associated with a same environment.

15. A non-transitory computer-readable medium having program code recorded thereon for controlling a robotic device, the program code executed by a processor and comprising:
program code to position the robotic device within a task environment;
program code to map a plurality of task image pixel descriptors associated with a first group of pixels in a task image of a scene in the task environment to a plurality of teaching image pixel descriptors associated with a second group of pixels in a teaching image of a teaching environment, each task image pixel descriptor of the plurality of task image pixel descriptors comprising a first pixel value associated with a pixel in the first group of pixels, and each teaching image pixel descriptor of the plurality of teaching image pixel descriptors comprising a second pixel value associated with a pixel in the second group of pixels, the first pixel value associated with each task image pixel descriptor having a same value as the second pixel value associated with the teaching image pixel descriptor mapped to the respective task image pixel descriptor;
program code to define a relative transform between the task image and the teaching image based on mapping the plurality of task image pixel descriptors, the relative transform indicating a change in an x-axis, y-axis, z-axis, roll, pitch, and yaw between the task image and the teaching image; and
program code to update parameters of a set of parameterized behaviors based on the relative transform to perform a task corresponding to the teaching image.

16. The non-transitory computer-readable medium of claim 15 , in which the set of parameterized behaviors comprises behaviors performed by a user while the robotic device was trained to perform the task with a virtual reality interface.

17. The non-transitory computer-readable medium of claim 15 , in which the plurality of task image pixel descriptors are mapped to the plurality of teaching image pixel descriptors at an interval.

18. The non-transitory computer-readable medium of claim 15 , in which:
a first position of the robotic device in the task environment is different from a second position of the robotic device during training in the teaching environment.

19. The non-transitory computer-readable medium of claim 18 , in which the first position and the second position are associated with one or both of a starting location or pose of the robotic device.

20. The non-transitory computer-readable medium of claim 18 , in

Assignees

• Toyota Res Inst Inc

Inventors

• Ma Jeremy
• Petersen Josh
• Nagarajan Umashankar
• Laskey Michael
• Helmick Daniel
• Borders James
• Shankar Krishna
• Stone Kevin
• Bajracharya Max

Classifications

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/09

  Supervised learning · CPC title

• B25J9/1666

  Avoiding collision or forbidden zones · CPC title

• G06V20/10 Primary

  Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59) · CPC title

• G06T7/248 Primary

  involving reference images or patches · CPC title