Training methods for deep networks

Patent metadata

Field	Value
Publication number	US-11113526-B2
Application number	US-201916570813-A
Country	US
Kind code	B2
Filing date	Sep 13, 2019
Priority date	Jul 23, 2019
Publication date	Sep 7, 2021
Grant date	Sep 7, 2021

Abstract

A method for training a deep neural network of a robotic device is described. The method includes constructing a 3D model using images captured via a 3D camera of the robotic device in a training environment. The method also includes generating pairs of 3D images from the 3D model by artificially adjusting parameters of the training environment to form manipulated images using the deep neural network. The method further includes processing the pairs of 3D images to form a reference image including embedded descriptors of common objects between the pairs of 3D images. The method also includes using the reference image from training of the neural network to determine correlations to identify detected objects in future images.

First claim

What is claimed is:

1. A method for training a deep neural network of a robotic device, comprising:
constructing a 3D model using images captured via a 3D camera of the robotic device in a training environment;
generating pairs of 3D images from the 3D model by artificially adjusting parameters of the training environment to form manipulated images using the deep neural network;
processing the pairs of 3D images to form a reference image including embedded descriptors of common objects between the pairs of 3D images; and
using the reference image from training of the neural network to determine correlations to identify detected objects in future images by:
overlaying the corresponding reference image over a captured image of a scene, and
determining an identity of the detected object based on a point correspondence between the corresponding reference image and the captured image and the embedded descriptors of the corresponding reference image, in which the embedded descriptors encode information into a series of numbers to provide a numerical fingerprint to differentiate one feature from another.

2. The method of claim 1 , in which generating the pairs of 3D images comprises:
pairing 3D images with linked elements; and
manipulating linked elements between the pairs of 3D images to create a scene having different object articulations.

3. The method of claim 1 , in which artificially adjusting parameters comprises:
varying an object articulation between an original 3D image and a manipulated 3D image.

4. The method of claim 3 , in which varying the object articulation comprises:
varying lighting between the original 3D image and the manipulated 3D image.

5. The method of claim 3 , in which varying the object articulation comprises:
varying a viewing angle between the original 3D image and the manipulated 3D image.

6. The method of claim 1 , further comprising:
identifying objects in an unknown environment that may be manipulated, regardless of deformation, object articulation, angle, and lighting in the unknown environment; and
manipulating an identified object.

7. A method for controlling a robotic device based on identification of detected objects in an unknown environment, comprising:
detecting an object in the unknown environment;
selecting a corresponding reference image including embedded descriptors corresponding to trained objects manipulated according to artificially adjusting parameters of an image capture environment; and
identifying the detected object according to the embedded descriptors of the corresponding reference image by:
overlaying the corresponding reference image over a captured image of a scene, and
determining an identity of the detected object based on a point correspondence between the corresponding reference image and the captured image and the embedded descriptors of the corresponding reference image, in which the embedded descriptors encode information into a series of numbers to provide a numerical fingerprint to differentiate one feature from another.

8. The method of claim 7 , further comprising tracking an identified object over a period of time.

9. The method of claim 7 , further comprising:
determining an identified object may be manipulated; and
manipulating the identified object.

10. A non-transitory computer-readable medium having program code recorded thereon for training a deep neural network of a robotic device, the program code being executed by a processor and comprising:
program code to generate pairs of 3D images from the 3D model by artificially adjusting parameters of a training environment to form manipulated images using the deep neural network;
program code to process the pairs of 3D images to form a reference image including embedded descriptors of common objects between the pairs of 3D images;
program code to use the reference image from training of the neural network to determine correlations to identify detected objects in future images by:
program code to overlay the corresponding reference image over a captured image of a scene, and
program code to determine an identity of a detected object based on a point correspondence between the corresponding reference image and the captured image and the embedded descriptors of the corresponding reference image, in which the embedded descriptors encode information into a series of numbers to provide a numerical fingerprint to differentiate one feature from another.

11. The non-transitory computer-readable medium of claim 10 , in which the program code to generate the pairs of 3D images comprises:
program code to pair 3D images with linked elements; and
program code to manipulate linked elements between the pairs of 3D images to create a scene having different object articulations.

12. The non-transitory computer-readable medium of claim 10 , in which the program code to generate the pairs of 3D images comprises:
program code to vary an object articulation between an original 3D image and a manipulated 3D image.

13. The non-transitory computer-readable medium of claim 12 , in which the program code to vary the object articulation comprises:
program code to vary lighting between the original 3D image and the manipulated 3D image.

14. The non-transitory computer-readable medium of claim 12 , in which the program code to vary the object articulation comprises:
program code to vary a viewing angle between the original 3D image and the manipulated 3D image.

15. A system for controlling a robotic device based on identification of detected objects in an unknown environment, the system comprising:
a pre-trained, object identification module to select a corresponding reference image to identify a detected object in a captured image, the corresponding reference image including embedded descriptors based on trained objects manipulated according to artificially adjusted parameters of an image capture environment, the pre-trained, object identification module to overlay the corresponding reference image over the captured image of a scene, and to determine an identity of the detected object based on a point correspondence between the corresponding reference image and the captured image and the embedded descriptors of the corresponding reference image, in which the embedded descriptors encode information into a series of numbers to provide a numerical fingerprint to differentiate one feature from another; and
a controller to select an autonomous behavior of the robotic device based on an identity of the detected object.

16. The system of claim 15 , in which the pre-trained, object identification module to track an identified object over a period of time.

17. The system of claim 15 , in which the controller is further to manipulate an identified object.

18. The system of claim 15 , in which the pre-trained, object identification module to detect common objects between the corresponding reference image and the captured image based on correlations to identify the detected object in future images.

Assignees

• Toyota Res Inst Inc

Inventors

• Stone Kevin
• Shankar Krishna
• Laskey Michael

Classifications

• G06V10/774

  Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06V10/82

  using neural networks · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06F18/28

  Determining representative reference patterns, e.g. by averaging or distorting; Generating dictionaries · CPC title

• G06F18/214

  Generating training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title