Vehicle scenario mining for machine learning models

Patent metadata

Field	Value
Publication number	US-11550851-B1
Application number	US-202217669186-A
Country	US
Kind code	B1
Filing date	Feb 10, 2022
Priority date	Feb 10, 2022
Publication date	Jan 10, 2023
Grant date	Jan 10, 2023

Abstract

Provided are methods for vehicle scenario mining for machine learning methods, which can include determining a set of attributes associated with an untested scenario for which a machine learning model of an autonomous vehicle is to make planned movements. The method includes searching a scenario database for the untested scenario based on the set of attributes. The scenario database includes a plurality of datasets representative of data received from an autonomous vehicle sensor system in which the plurality of datasets is marked with at least one attribute of the set of attributes. The method further includes obtaining the untested scenario from the scenario database for inputting into the machine learning model for training the machine learning model. The machine learning model is configured to make the planned movements for the autonomous vehicle. Systems and computer program products are also provided.

First claim

What is claimed is:

1. A method comprising:
determining, using at least one processor, a set of attributes associated with an untested scenario for which untested planned movements are uncertain and have not yet been generated by a machine learning model of a vehicle;
performing, using the at least one processor, a query at a scenario database based on the set of attributes associated with the untested scenario, the scenario database including a plurality of labeled vehicle sensor datasets representative of labeled vehicle sensor data received from a vehicle sensor system, the plurality of labeled vehicle sensor datasets having a subset of the labeled vehicle sensor datasets labeled with at least one attribute of the set of attributes, the set of attributes being indicative of the untested scenario in which the untested planned movements of the machine learning model are uncertain;
identifying, using the at least one processor and based on the query, the untested scenario at the scenario database from the plurality of labeled vehicle sensor datasets; and
obtaining, using the at least one processor, the untested scenario from the scenario database for inputting into the machine learning model for training the machine learning model, the machine learning model configured to generate the untested planned movements for the autonomous vehicle.

2. The method of claim 1 , further comprising:
constructing, using the at least one processor and in response to determining the set of attributes associated with the untested scenario, a search string based on the set of attributes, the search string configured to perform the query at the scenario database and identify the untested scenario having the at least one attribute of the set of attributes.

3. The method of claim 1 , wherein the scenario database comprises an SQL database, wherein the query comprises an SQL query, and wherein the scenario database is configured to carry out the SQL query.

4. The method of claim 1 , wherein determining the set of attributes associated with the untested scenario that the autonomous vehicle is to navigate further comprises:
determining, using the at least one processor, a first potential attribute frequency based on a first number of tested planned movements including a first potential attribute, the tested planned movements generated by the machine learning model for the vehicle;
determining, using the at least one processor, a second potential attribute frequency based on a second number of tested planned movements including a second potential attribute, the tested planned movements generated by the machine learning model for the vehicle;
determining, using the at least one processor, the second potential attribute frequency is lower than the first potential attribute frequency;
selecting, using the at least one processor, the second potential attribute to add to the set of attributes based on the second potential attribute frequency being lower than the first potential attribute frequency; and
adding, using the at least one processor, the second potential attribute to the set of attributes associated with the untested scenario for which the untested planned movements are uncertain and have not yet been generated by the machine learning model of the vehicle,
wherein the subset of the labeled vehicle sensor datasets is labeled with the second potential attribute of the set of attributes.

5. The method of claim 1 , wherein determining the set of attributes associated with the untested scenario that the autonomous vehicle is to navigate further comprises:
determining, by the one or more processors, a first potential attribute difficulty for the machine learning model to generate the planned movements in response to scenarios having a first potential attribute;
determining, using the at least one processor, a second potential attribute difficulty for the machine learning model to generate the planned movements in response to scenarios having a second potential attribute;
determining, using the at least one processor, the second potential attribute difficulty is greater than the first potential attribute difficulty;
selecting, using the at least one processor, the second potential attribute to add to the set of attributes based on the second potential attribute difficulty being greater than the first potential attribute difficulty; and
adding, using the least one processor, the second potential attribute to the set of attributes associated with the untested scenario for which the untested planned movements are uncertain and have not yet been generated by the machine learning model of the vehicle,
wherein the subset of labeled vehicle sensor datasets is labeled with the second potential attribute of the set of attributes.

6. The method of claim 1 , further comprising:
presenting, using the at least one processor, the untested scenario to the machine learning model for training the machine learning model, the machine learning model configured to generate the planned movements for the autonomous vehicle,
wherein the machine learning model was previously trained with at least one vehicle sensor dataset of the labeled vehicle sensor datasets and wherein the machine learning model has not yet generated planned movements in response to the untested scenario having the set of attributes.

7. The method of claim 1 , wherein the at least one attribute includes at least one of an agent-level attribute representative of a moving obstacle proximate to the autonomous vehicle, a scene-level attribute representative of environmental obstacle proximate to the autonomous vehicle, and an ego-level attribute representative of a characteristic of the autonomous vehicle, and
wherein the agent-level attribute is selected from at least one of a large truck, a motorcycle, a scooter, a class 1-8 vehicle, a cargo van, a bicycle, a large animal, or a pedestrian,
wherein the scene-level attribute is selected from at least one of an elevation, a hill steepness, a construction zone, a crosswalk, a stoplight, an HOV lane, a median, a traffic speed, a traffic volume, a number of vehicular and cyclist traffic lanes, a lane width, lane traffic directions, lane marker types, rainy conditions, snowy conditions, fog, and thunderstorms, a parked vehicle, an object in a roadway, an upcoming intersection, traffic conditions, roadway conditions, construction conditions, intersection conditions, pedestrians, or an emergency siren, and
wherein the ego-level attribute is selected from a presence of a sensor in the plurality of labeled vehicle sensor datasets or a value representative of data gathered by the sensor in the plurality of labeled vehicle sensor datasets.

8. The method of claim 1 , wherein the plurality of labeled vehicle sensor datasets are organized into a plurality of frames, wherein each frame of the plurality of frames has a time stamp and labeled attribute metadata, the labeled attribute metadata based on the labeled vehicle sensor data received from the vehicle sensor system corresponding to the time stamp.

9. The method of claim 8 , wherein obtaining the untested scenario further comprises:
obtaining, using the at least one processor, the plurality of frames having the labeled attribute metadata associated with the at least one attribute of the set of attributes.

10. The method of claim 1 , wherein the labeled vehicle sensor data received from the vehicle sensor system includes labeled vehicle sensor data related to an environment of the vehicle, and wherein each labeled vehicle sensor dataset of the plurality of labeled vehicle sensor datasets is marked with the at least one attribute of the set of attributes,
wherein the vehicle sensor system includ

Assignees

• Motional Ad Llc

Inventors

• Kabzan Juraj
• Gomes Julia

Classifications

• B60W60/001

  Planning or execution of driving tasks · CPC title

• G05B13/0265

  the criterion being a learning criterion · CPC title

• G06F16/90344 Primary

  by using string matching techniques · CPC title

• B60W2554/40

  Dynamic objects, e.g. animals, windblown objects · CPC title

• G06N20/00 Primary

  Machine learning · CPC title