Generative adversarial network enriched driving simulation

Patent metadata

Field	Value
Publication number	US-10768629-B2
Application number	US-201816043706-A
Country	US
Kind code	B2
Filing date	Jul 24, 2018
Priority date	Jul 24, 2018
Publication date	Sep 8, 2020
Grant date	Sep 8, 2020

Abstract

A computer-implemented method and a system for training a computer-based autonomous driving model used for an autonomous driving operation by an autonomous vehicle are described. The method includes: creating time-dependent three-dimensional (3D) traffic environment data using at least one of real traffic element data and simulated traffic element data; creating simulated time-dependent 3D traffic environmental data by applying a time-dependent 3D generic adversarial network (GAN) model to the created time-dependent 3D traffic environment data; and training a computer-based autonomous driving model using the simulated time-dependent 3D traffic environmental data.

First claim

What is claimed is:

1. A computer-implemented method comprising:
creating time-dependent three-dimensional (3D) traffic environment data using at least one of real traffic element data and simulated traffic element data;
creating simulated time-dependent 3D traffic environmental data by applying a time-dependent 3D generative adversarial network (GAN) model to the created time-dependent 3D traffic environment data, wherein the simulated time-dependent 3D traffic environmental data includes object movement data indicating an irregular movement of one or more objects around a road; and
training a computer-based autonomous driving model using the simulated time-dependent 3D traffic environmental data.

2. The computer-implemented method of claim 1 , further comprising:
creating the time-dependent 3D generative adversarial network (GAN) model through an adversarial machine learning process of a time-dependent 3D GAN discriminator sub-model and a time-dependent 3D GAN generator sub-model of the time-dependent 3D GAN model.

3. The computer-implemented method of claim 2 , wherein the adversarial machine learning process of the time-dependent 3D GAN discriminator sub-model comprises:
receiving time-dependent 3D GAN discriminator training data from the time-dependent 3D GAN generator sub-model;
performing, using the time-dependent 3D GAN discriminator sub-model, discrimination analysis of the received time-dependent 3D GAN discriminator training data to generate a discrimination result indicating whether the time-dependent 3D GAN discriminator sub-model determined that the time-dependent 3D GAN discriminator training data represents real-world time-dependent 3D traffic environmental data or simulated time-dependent 3D traffic environmental data;
performing matching of the generated discrimination result with supervisory data indicating whether the time-dependent 3D GAN discriminator training data represents real-world time-dependent 3D traffic environmental data or simulated time-dependent 3D traffic environmental data, to generate a training result indicating a trained level of the time-dependent 3D GAN discriminator sub-model; and
modifying parameter values of the time-dependent 3D GAN discrimination sub-model based on the training result.

4. The computer-implemented method of claim 3 , wherein the adversarial machine learning process of the time-dependent 3D GAN generator sub-model further comprises:
generating, using the time-dependent 3D GAN generator sub-model, simulated time-dependent 3D traffic environmental data;
providing the generated simulated time-dependent 3D traffic environmental data for creating time-dependent 3D GAN discriminator training data to be used by the time-dependent 3D GAN discriminator sub-model;
receiving the training result from the time-dependent 3D GAN discriminator sub-model; and
modifying parameter values of the time-dependent 3D GAN generator sub-model based on the training result.

5. The computer-implemented method of claim 2 , wherein the adversarial machine learning process of the time-dependent 3D GAN generator sub-model comprises:
generating, using the time-dependent 3D GAN generator sub-model, simulated time-dependent 3D traffic environmental data;
providing the generated simulated time-dependent 3D traffic environmental data for creating time-dependent 3D GAN discriminator training data;
receiving a training result indicating a trained level of the time-dependent 3D GAN discriminator sub-model from the time-dependent 3D GAN discriminator sub-model; and
modifying parameter values of the time-dependent 3D GAN generator sub-model based on the training result.

6. The computer-implemented method of claim 1 , wherein the training the computer-based autonomous driving model comprises:
rendering the simulated time-dependent 3D traffic environmental data to generate a virtual photorealistic time-dependent 3D traffic environment;
carrying out, using the computer-based autonomous driving model, a virtual autonomous driving operation in the generated virtual photorealistic time-dependent 3D traffic environment;
obtaining a virtual autonomous driving result of the virtual autonomous driving operation; and
modifying parameter values of the computer-based autonomous driving model based on the virtual autonomous driving result.

7. The computer-implemented method of claim 1 , further comprising performing a real-world autonomous driving operation using the trained computer-based autonomous driving model.

8. A system for an autonomous-driving vehicle, comprising:
one or more processors; and
memory storing instructions that, when executed by the one or more processors, cause the one or more processors to:
create time-dependent three-dimensional (3D) traffic environment data using at least one of real traffic element data and simulated traffic element data;
create simulated time-dependent 3D traffic environmental data by applying a time-dependent 3D generative adversarial network (GAN) model to the created time-dependent 3D traffic environment data, wherein the simulated time-dependent 3D traffic environmental data includes object movement data indicating an irregular movement of one or more objects around a road; and
train a computer-based autonomous driving model using the simulated time-dependent 3D traffic environmental data.

9. The computer-implemented method of claim 1 , wherein the real traffic element data is used to create the time-dependent 3D traffic environment data, and the real traffic element data include at least one of geographical mapping data, traffic sign data, and traffic signal data of a real-world geographical region.

10. The computer-implemented method of claim 1 , wherein the simulated traffic element data is used to create the time-dependent 3D traffic environment data, and the simulated traffic element data include at least one of simulated weather data, simulated traffic signal change data, simulated pedestrian data, and simulated obstacles data.

11. The system of claim 8 , wherein the simulated traffic element data is used to create the time-dependent 3D traffic environment data, and the simulated traffic element data include at least one of simulated weather data, simulated traffic signal change data, simulated pedestrian data, and simulated obstacles data.

12. The system of claim 8 , wherein the instructions cause the one or more processors to create the time-dependent 3D generative adversarial network (GAN) model through an adversarial machine learning process of a time-dependent 3D GAN discriminator sub-model and a time-dependent 3D GAN generator sub-model of the time-dependent 3D GAN model.

13. The system of claim 12 , wherein the adversarial machine learning process of the time-dependent 3D GAN discriminator sub-model comprises:
receiving time-dependent 3D GAN discriminator training data from the time-dependent 3D GAN generator sub-model;
performing, using the time-dependent 3D GAN discriminator sub-model, discrimination analysis of the received time-dependent 3D GAN discriminator training data to generate a discrimination result indicating whether the time-dependent 3D GAN discriminator sub-model determined that the time-dependent 3D GAN discriminator training data represents real-world time-dependent 3D traffic environmental data or simulated time-dependent 3D traffic environmental data;
performing matching of the generated discrimination result with supervisory data indicating whether the time-dependent 3D GAN discriminator training data represents real-world time-dependent 3D traffic environmental data or simulated time-dependent 3D traffic environmental data, to generate a tr

Assignees

• Pony Ai Inc

Inventors

• Song Hao
• Peng Jun
• Deng Nengxiu
• Xiao Sinan
• Qin Tao
• Lou tiancheng
• Li Tianyi
• Yu Xiang
• Zhang Yubo

Classifications

• B60W50/06

  Improving the dynamic response of the control system, e.g. improving the speed of regulation or avoiding hunting or overshoot · CPC title

• G06N3/08 Primary

  Learning methods · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06N3/047

  Probabilistic or stochastic networks · CPC title

• G06N3/0475

  Generative networks · CPC title