Training, testing, and verifying autonomous machines using simulated environments

Patent metadata

Field	Value
Publication number	US-12182694-B2
Application number	US-202217898887-A
Country	US
Kind code	B2
Filing date	Aug 30, 2022
Priority date	Mar 27, 2018
Publication date	Dec 31, 2024
Grant date	Dec 31, 2024

Abstract

In various examples, physical sensor data may be generated by a vehicle in a real-world environment. The physical sensor data may be used to train deep neural networks (DNNs). The DNNs may then be tested in a simulated environment—in some examples using hardware configured for installation in a vehicle to execute an autonomous driving software stack—to control a virtual vehicle in the simulated environment or to otherwise test, verify, or validate the outputs of the DNNs. Prior to use by the DNNs, virtual sensor data generated by virtual sensors within the simulated environment may be encoded to a format consistent with the format of the physical sensor data generated by the vehicle.

First claim

What is claimed is:

1. A method comprising:
receiving, using a simulator component, simulation data representing a simulated environment corresponding to a virtual representation of an autonomous or semi-autonomous machine;
generating, using the simulator component and based at least on the simulation data, virtual sensor data representing the simulated environment perceived using at least one virtual sensor of the virtual representation of the autonomous or semi-autonomous machine within the simulated environment;
receiving, using a computing system of a hardware-in-the-loop (HIL) device and from the simulator component, the virtual sensor data, the HIL device including hardware corresponding to the autonomous or semi-autonomous machine;
applying the virtual sensor data to one or more machine learning models being executed using the computing system of the HIL device;
determining, using the one or more machine learning models being executed using the computing system of the HIL device and based at least on the virtual sensor data, at least one operation for the virtual representation; and
transmitting, using the computing system and to the simulator component, operative data representing the at least one operation for the virtual representation.

2. The method of claim 1 , wherein the virtual sensor data is generated using simulation software executed using one or more computing devices.

3. The method of claim 1 , wherein:
the computing system of the HIL device that includes the hardware corresponding to the autonomous or semi-autonomous machine is included in a vehicle simulator component; and
the vehicle simulator component communicates with the simulator component to receive the simulation data and transmit the operative data.

4. The method of claim 2 , wherein:
when integrated into an autonomous or semi-autonomous machine, the computing system communicates with one or more other components of the autonomous or semi-autonomous machine using at least one of a communication type or a communication protocol; and
during a simulation corresponding to the simulation data, the computing system communicates with the one or more computing devices using the at least one of the communication type or the communication protocol.

5. The method of claim 1 , further comprising:
generating encoded sensor data by encoding the virtual sensor data using a sensor data format that corresponds to real-world sensor data generated using a real-world sensor of the autonomous or semi-autonomous machine,
wherein the determining the at least one operation is based at least on the encoded sensor data.

6. The method of claim 1 , wherein the virtual sensor data includes a sensor data format that corresponds to real-world sensor data generated using a real-world sensor of the autonomous or semi-autonomous machine.

7. The method of claim 1 , further comprising:
generating, using the simulation data, additional virtual sensor data representing the simulated environment perceived by at least one additional virtual sensor of the virtual representation; and
determining, using the one or more machine learning models executed using the computing system of the HIL device, at least one additional operation for the virtual representation based at least on the additional virtual sensor data,
wherein the operative data further represents the at least one additional operation for the virtual representation.

8. The method of claim 1 , wherein the virtual representation comprises at least one of a virtual vehicle or a virtual robot that simulates the autonomous or semi-autonomous machine.

9. The method of claim 1 , wherein the simulation data includes virtual data and real-world data, and the simulated environment includes one or more representations corresponding to the virtual data and one or more representations corresponding to the real-world data.

10. A method comprising:
obtaining virtual sensor data representing a simulated environment perceived by at least one virtual sensor of a virtual representation of an autonomous or semi-autonomous machine within the simulated environment, the virtual sensor data corresponding to a first data format;
generating, based at least on the virtual sensor data corresponding to the first data format, encoded sensor data by encoding the virtual sensor data using a second data format that is associated with at least one of one or more machine learning models or a hardware-in-the-loop (HIL) device that includes hardware corresponding to the autonomous or semi-autonomous machine;
determining, using the HIL device and based at least on the one or more machine learning models, at least one operation for the virtual representation based at least on the encoded sensor data; and
transmitting operative data representing the at least one operation of the virtual representation.

11. The method of claim 10 , wherein:
the HIL device is included in a vehicle simulator component; and
the vehicle simulator component communicates with a simulation component to perform the transmitting of the operative data.

12. The method of claim 10 , wherein:
when in deployment within the autonomous or semi-autonomous machine, the hardware communicates with one or more other components of the autonomous or semi-autonomous machine using at least one of a communication type or a communication protocol; and
during simulation, the hardware communicates with one or more computing devices to obtain at least one of the virtual sensor data or encoded sensor data using the at least one of the communication type or the communication protocol.

13. The method of claim 10 , wherein the at least one of virtual sensor data or the encoded sensor data is generated using simulation software executed using a one or more computing devices.

14. The method of claim 10 , wherein the virtual representation corresponds to at least one of a virtual vehicle or a virtual robot that simulates the autonomous or semi-autonomous machine.

15. The method of claim 10 , wherein the simulated environment includes representations of virtual data augmented with representations of real-world data.

16. The method of claim 10 , wherein the second data format of the encoded sensor data is associated with a driving software stack that is executed using the HIL device that includes the hardware corresponding to the autonomous or semi-autonomous machine, the driving software stack including the one or more machine learning models.

17. The method of claim 10 , wherein:
the generating of the encoded sensor data by encoding the virtual sensor data is performed using a simulation component; and
the method further comprises receiving, using one or more computing devices associated with the HIL device, the virtual sensor data from the simulation component.

18. A computing device comprising one or more processing units to:
receive, from one or more computing devices, virtual sensor data representing a simulated environment perceived using at least one virtual sensor of a virtual representation of an autonomous or semi-autonomous machine within the simulated environment, the computing device being part of a hardware-in-the-loop (HIL) device that includes hardware for installation in the autonomous or semi-autonomous machine;
determine, using one or more machine learning models, at least one operation for the virtual representation based at least on the virtual sensor data; and
transmit, to at least one computing device of the one or more computing devices, operative data representing the at least one operation of

Assignees

• Nvidia Corp

Inventors

• Farabet Clement
• Zedlewski John
• Taylor Zachary
• Heinrich Greg
• Delaunay Claire
• Daly Mark
• Campbell Matthew
• Beeson Curtis
• Hicok Gary
• Cox Michael
• Lebaredian Rev
• Tamasi Tony
• Auld David

Classifications

• G06N3/063 Primary

  using electronic means · CPC title

• G06N3/08 Primary

  Learning methods · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/091

  Active learning · CPC title

• G06N3/0475

  Generative networks · CPC title