Autonomous task performance based on visual embeddings

Patent metadata

Field	Value
Publication number	US-11741701-B2
Application number	US-202217667217-A
Country	US
Kind code	B2
Filing date	Feb 8, 2022
Priority date	Jul 23, 2019
Publication date	Aug 29, 2023
Grant date	Aug 29, 2023

Abstract

A method for controlling a robotic device is presented. The method includes capturing an image corresponding to a current view of the robotic device. The method also includes identifying a keyframe image comprising a first set of pixels matching a second set of pixels of the image. The method further includes performing, by the robotic device, a task corresponding to the keyframe image.

First claim

What is claimed is:

1. A method for controlling a robotic device, comprising:
capturing an image corresponding to a current view of the robotic device;
identifying a keyframe image based on matching one or more pixels of a first set of pixels associated with the keyframe image with one or more pixels of a second set of pixels associated with the image;
determining, based on identifying the keyframe, a difference between a first relationship of a keyframe robotic device to a first object in the keyframe image and a second relationship of the robotic device to a second object in the image;
adjusting a parameter associated with the task based on determining the difference between the first relationship and the second relationship, the task being associated with the keyframe image; and
controlling the robotic device to perform a task based on adjusting the parameter.

2. The method of claim 1 , further comprising capturing the keyframe image while training the robotic device to perform the task.

3. The method of claim 1 , in which the task comprises at least one of interacting with an object, navigating through an environment, or a combination thereof.

4. The method of claim 1 , in which:
the first relationship corresponds to a pose of the keyframe robotic device in relation to a first object in the keyframe image; and
the second relationship corresponds to a pose of the robotic device in relation to a second object in the image.

5. The method of claim 1 , in which:
the first relationship corresponds to a distance between the keyframe robotic device and a first object in the keyframe image; and
the second relationship corresponds to a distance between the robotic device and a second object in the image.

6. The method of claim 1 , in which:
each pixel of the first set of pixels and the second set of pixels is associated with a pixel descriptor; and
each pixel descriptor is associated with a set of values corresponding to pixel level information and depth information.

7. The method of claim 1 , in which the parameter is associated with one or more of a velocity or a position associated with the task.

8. An apparatus for controlling a robotic device, comprising:
a processor; and
a memory coupled with the processor and storing instructions operable, when executed by the processor, to cause the apparatus to:
capture an image corresponding to a current view of the robotic device;
identify a keyframe image based on matching one or more pixels of a first set of pixels associated with the keyframe image with one or more pixels of a second set of pixels associated with the image;
determine, based on identifying the keyframe, a difference between a first relationship of a keyframe robotic device to a first object in the keyframe image and a second relationship of the robotic device to a second object in the image;
adjust a parameter associated with the task based on determining the difference between the first relationship and the second relationship, the task being associated with the keyframe image; and
control the robotic device to perform a task based on adjusting the parameter.

9. The apparatus of claim 8 , in which execution of the instructions further cause the apparatus to capture the keyframe image while training the robotic device to perform the task.

10. The apparatus of claim 8 , in which the task comprises at least one of interacting with an object, navigating through an environment, or a combination thereof.

11. The apparatus of claim 8 , in which:
the first relationship corresponds to a pose of the keyframe robotic device in relation to a first object in the keyframe image; and
the second relationship corresponds to a pose of the robotic device in relation to a second object in the image.

12. The apparatus of claim 8 , in which:
the first relationship corresponds to a distance between the keyframe robotic device and a first object in the keyframe image; and
the second relationship corresponds to a distance between the robotic device and a second object in the image.

13. The apparatus of claim 8 , in which:
each pixel of the first set of pixels and the second set of pixels is associated with a pixel descriptor; and
each pixel descriptor is associated with a set of values corresponding to pixel level information and depth information.

14. The apparatus of claim 8 , in which the parameter is associated with one or more of a velocity or a position associated with the task.

15. A non-transitory computer-readable medium having program code recorded thereon for controlling a robotic device, the program code comprising:
program code to capture an image corresponding to a current view of the robotic device;
program code to identify a keyframe image based on matching one or more pixels of a first set of pixels associated with the keyframe image with one or more pixels of a second set of pixels associated with the image;
program code to determine, based on identifying the keyframe, a difference between a first relationship of a keyframe robotic device to a first object in the keyframe image and a second relationship of the robotic device to a second object in the image;
program code to adjust a parameter associated with the task based on determining the difference between the first relationship and the second relationship, the task being associated with the keyframe image; and
program code to control the robotic device to perform a task based on adjusting the parameter.

16. The non-transitory computer-readable medium of claim 15 , in which:
the first relationship corresponds to a pose of the keyframe robotic device in relation to a first object in the keyframe image; and
the second relationship corresponds to a pose of the robotic device in relation to a second object in the image.

17. The non-transitory computer-readable medium of claim 15 , in which:
the first relationship corresponds to a distance between the keyframe robotic device and a first object in the keyframe image; and
the second relationship corresponds to a distance between the robotic device and a second object in the image.

18. The non-transitory computer-readable medium of claim 15 , in which:
each pixel of the first set of pixels and the second set of pixels is associated with a pixel descriptor; and
each pixel descriptor is associated with a set of values corresponding to pixel level information and depth information.

19. The non-transitory computer-readable medium of claim 15 , in which the parameter is associated with one or more of a velocity or a position associated with the task.

Assignees

• Toyota Res Inst Inc

Inventors

• Ma Jeremy
• Stone Kevin
• Bajracharya Max
• Shankar Krishna

Classifications

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06V20/10 Primary

  Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59) · CPC title

• B25J9/1605 Primary

  Simulation of manipulator lay-out, design, modelling of manipulator · CPC title

• B25J9/1661

  characterised by task planning, object-oriented languages · CPC title