Adapting simulation data to real-world conditions encountered by physical processes

Patent metadata

Field	Value
Publication number	US-11273553-B2
Application number	US-201815995003-A
Country	US
Kind code	B2
Filing date	May 31, 2018
Priority date	Jun 5, 2017
Publication date	Mar 15, 2022
Grant date	Mar 15, 2022

Abstract

One embodiment of the present invention sets forth a technique for generating simulated training data for a physical process. The technique includes receiving, as input to at least one machine learning model, a first simulated image of a first object, wherein the at least one machine learning model includes mappings between simulated images generated from models of physical objects and real-world images of the physical objects. The technique also includes performing, by the at least one machine learning model, one or more operations on the first simulated image to generate a first augmented image of the first object. The technique further includes transmitting the first augmented image to a training pipeline for an additional machine learning model that controls a behavior of the physical process.

First claim

What is claimed is:

1. A method for generating simulated training data for a physical process, the method comprising:
receiving, as input to at least one machine learning model, a first simulated image of a first object, wherein the at least one machine learning model includes mappings between simulated images generated from models of physical objects and real-world images of the physical objects;
performing, by the at least one machine learning model, one or more operations on the first simulated image to generate a first augmented image of the first object; and
transmitting the first augmented image as training data to a training pipeline that trains an additional machine learning model to control a behavior of the physical process.

2. The method of claim 1 , wherein receiving the first simulated image of the first object comprises generating the first simulated image from a computer aided design (CAD) model of the first object.

3. The method of claim 1 , further comprising:
generating simulated training data that comprises the simulated images and real-world training data that comprises the real-world images; and
inputting the simulated training data and the real-world training data as unpaired training data for training the at least one machine learning model.

4. The method of claim 1 , further comprising:
generating labels associated with the first simulated image; and
transmitting the labels and the first augmented image as training data to the training pipeline.

5. The method of claim 4 , wherein the labels comprise a type of the first object, a graspable point on the first object, a position of the first object in the first augmented image, and an orientation of the first object in the first augmented image.

6. The method of claim 1 , wherein the additional machine learning model comprises an artificial neural network.

7. The method of claim 1 , wherein the at least one machine learning model comprises a generator neural network that produces augmented images from simulated images.

8. The method of claim 7 , wherein the at least one machine learning model further comprise a discriminator neural network that categorizes augmented images produced by the generator network as simulated or real.

9. The method of claim 1 , wherein the one or more operations performed by the at least one machine learning model comprise at least one of:
performing one or more shading operations on the first simulated image;
performing one or more lighting operations on the first simulated image; and
performing one or more operations that add noise to the first simulated image.

10. The method of claim 1 , wherein the physical process comprises a robot performing a grasping task.

11. One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of:
receiving, as input to at least one machine learning model, a first simulated image of a first object, wherein the at least one machine learning model includes mappings between simulated images generated from models of physical objects and real-world images of the physical objects;
performing, by the at least one machine learning model, one or more operations on the first simulated image to generate a first augmented image of the first object; and
transmitting the first augmented image as training data to a training pipeline that trains an additional machine learning model to control a behavior of the physical process.

12. The one or more non-transitory computer-readable media of claim 11 , wherein the method further comprises:
generating simulated training data that comprises the simulated images and real-world training data that comprises the real-world images; and
inputting the simulated training data and the real-world training data as unpaired training data for training the at least one machine learning model.

13. The one or more non-transitory computer-readable media of claim 11 , wherein the method further comprises:
generating labels associated with the first simulated image; and
transmitting the labels and the first augmented image as training data to the training pipeline.

14. The one or more non-transitory computer-readable media of claim 11 , wherein the first simulated image and the first augmented image comprise at least one of:
a two-dimensional (2D) representation of the first object; and
one or more three-dimensional (3D) locations associated with the first object.

15. The one or more non-transitory computer-readable media of claim 11 , wherein the method further comprises:
performing, by the at least one machine learning model, the one or more operations on a second simulated image of a second object to generate a second augmented image of the second object; and
transmitting the second augmented image to the training pipeline.

16. The one or more non-transitory computer-readable media of claim 11 , wherein the at least one machine learning model comprises:
a generator neural network that produces augmented images from simulated images; and
a discriminator neural network that categorizes augmented images produced by the generator network as simulated or real.

17. The one or more non-transitory computer-readable media of claim 11 , wherein the additional machine learning model comprises an artificial neural network.

18. The one or more non-transitory computer-readable media of claim 11 , wherein the one or more operations performed by the at least one machine learning model comprise at least one of:
performing one or more shading operations on the first simulated image;
performing one or more lighting operations on the first simulated image; and
performing one or more operations that add noise to the first simulated image.

19. The one or more non-transitory computer-readable media of claim 11 , wherein the physical process comprises a robot performing a grasping task.

20. A system, comprising:
a memory that stores instructions, and
a processor that is coupled to the memory and, when executing the instructions, is configured to:
receive, as input to at least one machine learning model, a first simulated image of a first object, wherein the at least one machine learning model includes mappings between simulated images generated from models of physical objects and real-world images of the physical objects;
perform, by the at least one machine learning model, one or more operations on the first simulated image to generate a first augmented image of the first object; and
transmit the first augmented image as training data to a training pipeline that trains an additional machine learning model to control a behavior of the physical process.

Assignees

• Autodesk Inc

Inventors

• Li Hui
• Atherton Evan Patrick
• Bradner Erin
• Cote Nicholas
• Kerrick Heather

Classifications

• G06N3/044

  Recurrent networks, e.g. Hopfield networks · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/0442

  characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] · CPC title

• G06N3/0475

  Generative networks · CPC title

• G06N3/094

  Adversarial learning · CPC title