In-vehicle operation of simulation scenarios during autonomous vehicle runs

Patent metadata

Field	Value
Publication number	US-11648959-B2
Application number	US-202017074807-A
Country	US
Kind code	B2
Filing date	Oct 20, 2020
Priority date	Oct 20, 2020
Publication date	May 16, 2023
Grant date	May 16, 2023

Abstract

This document describes methods by which an autonomous vehicle deploys a simulation scenario while operating in a real-world environment. The vehicle's sensors collect perception data. During a run of the vehicle in a real-world environment, the vehicle's on-board computing system will: (i) receive the perception data; (ii) publish the perception data to a muxing tool of the on-board computing system; (iii) generate simulation data that identifies and labels one or more simulated objects in the environment; (iv) publish the simulation data to the muxing tool; (v) use the muxing tool to add at least a portion of the simulation data to the perception data to yield augmented perception data; and (v) use at least a portion of the augmented perception data to make one or more navigation decisions based for the autonomous vehicle.

First claim

The invention claimed is:

1. A method of deploying a simulation scenario during an autonomous vehicle run, the method comprising:
by an on-board computing system of an autonomous vehicle during a run of the autonomous vehicle in a real-world environment:
receiving perception data from one or more vehicle sensors, wherein the perception data identifies and labels each of one or more perceived objects in the real-world environment;
publishing the perception data to a muxing tool of the on-board computing system;
detecting an occurrence of a trigger event;
enabling simulation operations in response to detection of the occurrence of the trigger event, the simulation operations comprising selection of a simulation scenario from a plurality of simulation scenarios and performance of the selected simulation scenario to generate simulation perception data that identifies and labels each of one or more simulated objects representing possible objects that the autonomous vehicle could perceive in the real-world environment;
identifying at least one said simulated object of the simulation perception data that is duplicative of an object in the perception data or has a same location in a scene as the object of the perception data;
modifying the simulation perception data to remove data therefrom that is associated with the at least one said simulated object which was identified;
publishing the simulation perception data to the muxing tool;
using the muxing tool to add at least a portion of the simulation perception data to the perception data to yield augmented perception data; and
using at least a portion of the augmented perception data to make one or more navigation decisions based for the autonomous vehicle.

2. The method of claim 1 , further comprising, by the on-board computing system before the muxing tool adds at least a portion of the simulation perception data to the perception data:
determining whether a conflict exists between a portion of the simulation perception data and a portion of the perception data to which the portion of the simulation perception data would be added;
if the conflict can be corrected, correcting the conflict; and
if the conflict cannot be corrected, discarding the portion of the simulation perception data.

3. The method of claim 2 , wherein the at least one said simulated object of the simulation perception data is identified as being duplicative of the object of the perception data based on an analysis of:
labels assigned to the perceived objects and
labels assigned to the simulated objects.

4. The method of claim 2 , further comprising:
receiving, from one or more sensors of the autonomous vehicle, vehicle characteristic data indicating one or more of the following: location, pose, speed, or trajectory of the autonomous vehicle; and
wherein the portion of the simulation perception data that is discarded comprises at least one said simulated object that was generated via the selected simulation scenario having vehicle characteristic requirements that conflict with the received vehicle characteristic data.

5. The method of claim 1 , wherein:
the perception data for each of the perceived objects includes an identifier and a label;
the simulation perception data for each of the simulated objects includes an identifier and a label; and
the at least one said imulated object of the simulation perception data is identified as being duplicative of the object of the perception data when the label and the identifier of the at least one said simulated object of the simulation perception data respectively match the label and the identifier of the perceived object.

6. The method of claim 1 , further comprising:
monitoring location data as received by one or more sensors of the autonomous vehicle; and
only permitting the muxing tool to generate augmented perception data when the location data corresponds to a designated simulation zone.

7. The system of claim 1 , further comprising:
monitoring location data as received by a one or more sensors of the autonomous vehicle; and
detecting the occurrence of the trigger event when the location data indicates that the autonomous vehicle is currently located in a designated geographic location.

8. The system of claim 7 , further comprising resetting generation of the simulation perception data when the location data indicates that the autonomous vehicle is no longer within the designated geographic location.

9. The method of claim 1 , wherein the simulation scenario is selected from the plurality of simulation scenarios by:
obtaining an identifier for a map that the autonomous vehicle is using;
identifying at least one of the plurality of simulation scenarios which has an identifier that is compatible with the identifier for the map; and
selecting the identified simulation scenario to generate the simulation perception data.

10. The method of claim 1 , wherein:
the simulation perception data is also generated in part by a remote computing system that is external to the autonomous vehicle; and
the method further comprises, by the remote computing system:
publishing the simulation perception data to a fleet comprising the autonomous vehicle and one or more additional vehicles, and
receiving, from a plurality of the vehicles to which the simulation perception data was published, navigation decision data that is responsive to vehicle-specific augmented perception data.

11. A vehicle, comprising:
one or more sensors configured to collect data corresponding to objects;
an engine or motor configured to power movement of the vehicle; and
an on-board computing system comprising a processor and a memory device with programming instructions that are configured to cause the processor to, during a run of the vehicle in an real-world environment:
receive perception data from the one or more sensors, wherein the perception data identifies and labels each of one or more perceived objects in the real-world environment,
publish the perception data to a mixing tool of the on-board computing system,
detecting an occurrence of a trigger event,
enabling simulation operations in response to detection of the occurrence of the trigger event, the simulation operations comprising selection of a simulation scenario from a plurality of simulation scenarios and performance of the selected simulation scenario to generate simulation perception data that identifies and labels for each of the one or more simulated objects representing possible objects that the autonomous vehicle could perceive in the real-world environment,
identify at least one said simulated object of the simulation perception data that is duplicative of an object in the perception data or has a same location in a scene as the object of the perception data,
modifying the simulation perception data to remove data therefrom that is associated with the at least one said simulated object which was identified,
publish the simulation perception data to the mixing tool,
use the mixing tool to add at least a portion of the simulation perception data to the perception data to yield augmented perception data, and
use at least a portion of the augmented perception data to make one or more navigation decisions for movement of the vehicle.

12. The vehicle of claim 11 , further comprising additional programming instructions that are configured to cause the processor to, before the mixing tool adds at least a portion of the simulation perception data to the perception data:
determine whether a conflict exists between a portion of the simulation perception data and a portion of the perception data to which the portion of the simulation per

Assignees

• Argo Ai Llc

Inventors

• Ariyaratne Arjuna
• Ackenhausen Thomas Carl
• Carmody Patrick Michael

Classifications

• G06F11/3696

  Methods or tools to render software testable · CPC title

• G06F11/3684

  for test design, e.g. generating new test cases · CPC title

• G01C21/3804

  Creation or updating of map data · CPC title

• B60W60/001

  Planning or execution of driving tasks · CPC title

• G01C21/3461

  Preferred or disfavoured areas, e.g. dangerous zones, toll or emission zones, intersections, manoeuvre types or segments such as motorways, toll roads or ferries · CPC title