System and method for evaluating a trained vehicle data set familiarity of a driver assitance system

Patent metadata

Field	Value
Publication number	US-10839263-B2
Application number	US-201816156366-A
Country	US
Kind code	B2
Filing date	Oct 10, 2018
Priority date	Oct 10, 2018
Publication date	Nov 17, 2020
Grant date	Nov 17, 2020

Abstract

The present disclosure relates to systems, devices and methods for evaluating a trained vehicle data set of a driver assistance system. Embodiments are directed to scoring run time attributes of a scene detection operation using a trained vehicle data set against a vector representation for an annotated data set to assess the ability of the scene detection operation to perceive target object attributes of the vehicle sensor data. In one embodiment, scoring evaluates effectiveness of the scene detection operation in identifying target object attributes of the vehicle sensor data using the trained vehicle data set. An event flag may be determined for a trained vehicle data set based on the scoring, the even flag identifying one or more parameters for updating the trained vehicle data set. Configurations and processes can identify anomalies to a trained vehicle data set and allow for capturing useful real world data to test runtime operations.

First claim

What is claimed is:

1. A method for evaluating a trained vehicle data set of a driver assistance system, the method comprising:
receiving, by a control unit, vehicle sensor data captured by at least one sensor of a vehicle, the vehicle sensor data generated by a driver assistance system of the vehicle;
running, by the control unit, a scene detection operation on the vehicle sensor data using a trained vehicle data set to identify target object attributes of the vehicle sensor data;
scoring, by the control unit, run time attributes of the scene detection operation against a vector representation for an annotated data set to assess an ability of the scene detection operation to perceive the target object attributes of the vehicle sensor data using the trained vehicle data set, wherein scoring evaluates effectiveness of the scene detection operation in identifying the target object attributes of the vehicle sensor data;
determining, by the control unit, an event flag for the trained vehicle data set based on the scoring, whether the event flag identifies at least one of a parameter and data sample for updating the trained vehicle data set; and
responsive to determining the event flag, transmitting, by the control unit, a segment of the vehicle sensor data to one of a data storage unit and a remote server, the segment of the vehicle sensor data selected based on the event flag.

2. The method of claim 1 , wherein the vehicle sensor data includes at least one of image, radar, and LiDAR data for a detection zone of the driver assistance system of the vehicle.

3. The method of claim 1 , wherein running the scene detection operation on the vehicle sensor data identifies target objects in real time based on attributes of the trained vehicle data set, the trained vehicle data set providing a plurality of object types and object attributes.

4. The method of claim 1 , wherein the trained vehicle data set includes object types and object attributes for a known annotated data set.

5. The method of claim 1 , wherein the vector representation for the annotated data set provides a spatial representation for a plurality of objects and object attributes of ground truth objects and wherein the vector representation for the annotated data set is used as a reference for comparing the vehicle sensor data.

6. The method of claim 1 , wherein scoring run time attributes of the scene detection operation includes determining a value to represent object types and object attributes not included and underrepresented in the trained vehicle data set.

7. The method of claim 1 , wherein scoring run time attributes of the scene detection operation includes determining a measure of a known data set relative to instantaneous vehicle data during in a runtime environment, the scoring to include comparing handled events and anomalies detected.

8. The method of claim 1 , wherein an effectiveness score for the scene detection operation is determined based on a probability that the trained vehicle data set can handle target objects of the vehicle sensor data.

9. The method of claim 8 , wherein the segment of the vehicle sensor data is a subset of the vehicle sensor data for which the effectiveness score for the scene detection operation is below a threshold effectiveness score for the scene detection operation.

10. The method of claim 1 , further comprising:
receiving an updated annotated data set from the remote server based on the segment of the vehicle sensor data.

11. A vehicle control unit comprising:
an input configured to receive vehicle sensor data from at least one sensor of a vehicle; and
a control unit coupled to the input, wherein the control unit includes instructions stored in a data storage unit that, when executed, cause the control unit to:
receive the vehicle sensor data captured by the at least one sensor of the vehicle, the vehicle sensor data generated by a driver assistance system of the vehicle;
run a scene detection operation on the vehicle sensor data using a trained vehicle data set to identify target object attributes of the vehicle sensor data;
score run time attributes of the scene detection operation against a vector representation for an annotated data set to assess an ability of the scene detection operation to perceive the target object attributes of the vehicle sensor data using the trained vehicle data set, wherein scoring evaluates effectiveness of the scene detection operation in identifying the target object attributes of the vehicle sensor data;
determine an event flag for the trained vehicle data set based on the scoring, whether the event flag identifies a parameter for updating the trained vehicle data set; and
responsive to determining the event flag, transmit a segment of the vehicle sensor data to one of the data storage unit and a remote server, the segment of the vehicle sensor data selected based on the event flag.

12. The vehicle control unit of claim 11 , wherein the vehicle sensor data includes at least one of image, radar, and LiDAR data for a detection zone of the driver assistance system of the vehicle.

13. The vehicle control unit of claim 11 , wherein running the scene detection operation on the vehicle sensor data identifies target objects in real time based on attributes of the trained vehicle data set, the trained vehicle data set providing a plurality of object types and object attributes.

14. The vehicle control unit of claim 11 , wherein the trained vehicle data set includes object types and object attributes for a known annotated data set.

15. The vehicle control unit of claim 11 , wherein the vector representation for the annotated data set provides a spatial representation for a plurality of objects and object attributes of ground truth objects and wherein the vector representation for the annotated data set is used as a reference for comparing the vehicle sensor data.

16. The vehicle control unit of claim 11 , wherein scoring run time attributes of the scene detection operation includes determining a value to represent object types and object attributes not included and underrepresented in the trained vehicle data set.

17. The vehicle control unit of claim 11 , wherein scoring run time attributes of the scene detection operation includes determining a measure of a known data set relative to instantaneous vehicle data during in a runtime environment, the scoring to include comparing handled events and anomalies detected.

18. The vehicle control unit of claim 11 , wherein an effectiveness score for the scene detection operation is determined based on a probability that the trained vehicle data set can handle target objects of the vehicle sensor data.

19. The vehicle control unit of claim 18 , wherein the segment of the vehicle sensor data is a subset of the vehicle sensor data for which the effectiveness score for the scene detection operation is below a threshold effectiveness score for the scene detection operation.

20. The vehicle control unit of claim 11 , wherein the control unit includes further instructions stored in the data storage unit that, when executed, cause the control unit to:
receive an updated annotated data set from the remote server based on the segment of the vehicle sensor data.

Assignees

• Harman Int Ind

Inventors

• Thompson Aaron
• Tan Honghao

Classifications

• G06F18/214

  Generating training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06N20/00 Primary

  Machine learning · CPC title

• G06V20/56

  exterior to a vehicle by using sensors mounted on the vehicle · CPC title

• B60W2050/0088

  Adaptive recalibration · CPC title

• B60W2554/402

  Type · CPC title