Machine learning methods and apparatus for robotic manipulation and that utilize multi-task domain adaptation

Patent metadata

Field	Value
Publication number	US-10773382-B2
Application number	US-201815913212-A
Country	US
Kind code	B2
Filing date	Mar 6, 2018
Priority date	Sep 15, 2017
Publication date	Sep 15, 2020
Grant date	Sep 15, 2020

Abstract

Implementations are directed to training a machine learning model that, once trained, is used in performance of robotic grasping and/or other manipulation task(s) by a robot. The model can be trained using simulated training examples that are based on simulated data that is based on simulated robot(s) attempting simulated manipulations of various simulated objects. At least portions of the model can also be trained based on real training examples that are based on data from real-world physical robots attempting manipulations of various objects. The simulated training examples can be utilized to train the model to predict an output that can be utilized in a particular task—and the real training examples used to adapt at least a portion of the model to the real-world domain can be tailored to a distinct task. In some implementations, domain-adversarial similarity losses are determined during training, and utilized to regularize at least portion(s) of the model.

First claim

What is claimed is:

1. A method implemented by one or more processors, comprising:
identifying a simulated training example generated based on simulated data, the simulated data generated during performance of a simulated robotic manipulation episode in which a simulated robot manipulates a simulated object in a simulated environment, wherein the simulated training example includes:
simulated training example input, and
simulated training example output that is based on whether a first robotic manipulation task was successful for the simulated robotic manipulation episode;

identifying a real training example generated based on real data, the real data generated during performance of a real robotic manipulation episode in which a real robot manipulates a real object, wherein the real training example includes:
real training example input, and
real training example output that is based on whether a second robotic manipulation task was successful for the real robotic manipulation episode, the second robotic manipulation task being distinct from the first robotic manipulation task;

applying the simulated training example input to a machine learning model;
generating a first predicted output based on processing of the simulated training example input using the machine learning model;
generating a first loss based on comparing the first predicted output to the simulated training example output that is based on whether the first robotic manipulation task was successful;
applying the real training example input to the machine learning model, or to an additional machine learning model that includes a portion of the machine learning model;
generating a second predicted output based on processing of the real training example input using the machine learning model or the additional machine learning model;
generating a second loss based on comparing the second predicted output to the real training example output that is based on whether the second robotic manipulation task was successful; and
training at least the portion of the machine learning model based on both the first loss and the second loss.

2. The method of claim 1 , further comprising, after training at least the portion of the machine learning model based on both the first loss and the second loss:
using the machine learning model in control of an additional real robot in performance of the first robotic manipulation task.

3. The method of claim 2 , wherein using the machine learning model in control of the additional real robot comprises iteratively applying inputs to the machine learning model, generating iterative predictions based on the iterative applying, and servoing an end effector of the additional real robot based on the iterative predictions.

4. The method of claim 1 , further comprising:
generating a third loss based on intermediate output generated during generating the second predicted output;
wherein training at least the portion of the machine learning model is further based on the third loss.

5. The method of claim 4 , wherein training at least the portion of the machine learning model comprises using the third loss in determining an effect, of the second loss, in training at least the portion of the machine learning model based on the second loss.

6. The method of claim 5 , wherein the third loss is a domain-adversarial similarity loss.

7. The method of claim 4 , wherein the intermediate output is generated over the portion of the machine learning model, and wherein training at least the portion of the machine learning model comprises:
training only the portion of the machine learning model based on the second loss and the third loss; and
training an entirety of the machine learning model based on the first loss.

8. The method of claim 1 , wherein training at least the portion of the machine learning model based on the first loss comprises training an entirety of the machine learning model based on the first loss.

9. The method of claim 8 , wherein training at least the portion of the machine learning model based on the second loss comprises training only the portion of the machine learning model based on the second loss.

10. The method of claim 1 , wherein generating the second predicted output is based on processing of the real training example input using the additional machine learning model.

11. The method of claim 10 , wherein the real training example input has different dimensions than the simulated training example input, and the additional machine learning model has different input dimensions than the machine learning model.

12. The method of claim 10 , wherein the real training example input has fewer dimensions than the simulated training example input, and the additional machine learning model has fewer input dimensions than the machine learning model.

13. The method of claim 12 , wherein the machine learning model is an instance grasping model, and the additional machine learning model is an indiscriminate grasping model.

14. The method of claim 12 , wherein the machine learning model is a neural network model having one or more neural network layers that are not included in the additional machine learning model.

15. The method of claim 1 , wherein the first robotic manipulation task is an instance grasping task and wherein the second robotic manipulation task is an indiscriminate grasping task.

16. The method of claim 1 , wherein
the simulated training example input includes one or more of: at least one rendered image of the simulated environment during the performance of the simulated robotic manipulation episode, and a motion vector that is based on one or more poses of the simulated robot during the simulated robotic manipulation episode; and
the real training example input includes one or more of: at least one real image captured by a camera of the real robot during the performance of the real robotic manipulation episode, and a real motion vector that is based on one or more poses of the real robot during the real robotic manipulation episode.

17. The method of claim 1 , wherein the simulated training example input includes a segmentation mask tailored to a target object for the first robotic manipulation task for the simulated robotic manipulation episode, and wherein the real training example input lacks any segmentation mask, or includes a default segmentation mask that is not tailored to any object of the real robotic manipulation episode.

18. A method implemented by one or more processors, comprising:
generating a first predicted output based on processing of simulated training example input, of a simulated training example, using a machine learning model;
generating a first loss based on comparing the first predicted output to simulated training example output of the simulated training example, wherein the simulated training example output is for a first robotic manipulation task;
generating a second predicted output based on processing of real training example input, of a real training example, using the machine learning model or an additional machine learning model that includes a portion of the machine learning model;
generating a second loss based on comparing the second predicted output to real training example output of the real training example, wherein the real training example output is for a second robotic task that is distinct from the first robotic manipulation task; and
training at least the portion of the machine learning model based on both the first loss and the second loss.

19. The method of claim 1

Assignees

• X Dev Llc

Inventors

• Bai Yunfei
• Fang Kuan
• Hinterstoisser Stefan
• Kalakrishnan Mrinal

Classifications

• G05B2219/40514

  Computed robot optimized configurations to train ann, output path in real time · CPC title

• G05B2219/39316

  Two ann, second ann trained with calibration data to learn error first ann · CPC title

• G05B2219/39536

  Planning of hand motion, grasping · CPC title

• G05B2219/39543

  Recognize object and plan hand shapes in grasping movements · CPC title

• B25J9/1612

  characterised by the hand, wrist, grip control · CPC title