Fast acquisition of labeled vehicular data from multiple data sources

Patent metadata

Field	Value
Publication number	US-11295180-B1
Application number	US-202017064428-A
Country	US
Kind code	B1
Filing date	Oct 6, 2020
Priority date	Oct 6, 2020
Publication date	Apr 5, 2022
Grant date	Apr 5, 2022

Abstract

Approaches, techniques, and mechanisms are disclosed for generating assisted driving test and evaluation data. According to one embodiment, video and non-video data are collected asynchronously through video and non-video data interface from an image acquisition device and a non-video data source collocated with the vehicle. Timing information received with the data is used to synchronize the video and non-video data into synchronized vehicle data. Specific labels indicating ground truths are identified to be applied to the synchronized vehicle data. Labeled vehicular data is generated from the synchronized vehicle data and the specific labels. At least a portion of the labeled vehicular data is to be used to test and evaluate assisted driving functionality.

First claim

What is claimed is:

1. A method comprising:
collecting, asynchronously, video data through a video data interface from an image acquisition device and non-video data through a non-video data interface from a non-video data source, the image acquisition device and the non-video data source being collocated with a vehicle;
using first timing information received with the video data and second timing information received with the non-video data to synchronize the video data and the non-video data into synchronized vehicle data, datasets of the synchronized vehicle data having a common timeline, the common timeline indicating individual time points to which the datasets of the synchronized vehicle data respectively correspond, wherein the first timing information is generated by a video encoder of the image acquisition device;
identifying specific labels to be applied to the synchronized vehicle data, the labels indicating ground truths for the synchronized vehicle data; and
generating, by one or more computing devices collocated with the vehicle, labeled vehicular data from the synchronized vehicle data and the specific labels.

2. The method of claim 1 , wherein at least a portion of the labeled vehicular data is used for testing and evaluation of assisted driving functionality of the vehicle and wherein the assisted driving functionality includes generating predictions for driving decisions in different driving conditions.

3. The method of claim 1 , wherein the image acquisition device is at least one from a group including a mobile phone camera, an external camera attached to a device collocated in the vehicle, a LIDAR device, and combinations thereof.

4. The method of claim 1 , wherein the non-video data source represents a data source communicated through one or more from a group including: FlexRay protocols, CAN protocols, TCP/IP protocols, USB protocols, BlueRay protocols, Wi-Fi protocols, and combinations thereof.

5. The method of claim 1 , wherein the labels comprise one or more of: manual tagging labels, or automatic tagging labels.

6. The method of claim 5 , wherein the automatic tagging labels comprise one or more labels automatically generated by neural networks implemented on an end user device collocated with the vehicle.

7. The method of claim 5 , wherein the manual tagging labels comprise one or more labels determined at least in part from user input received through a user interface.

8. The method of claim 1 , wherein the one or more computing devices is at least one mobile device and wherein the at least one mobile device performs the collecting, synchronizing and identifying steps.

9. One or more non-transitory computer readable media storing a program of instructions that is executable by one or more computing processors to perform:
collecting, asynchronously, video data through a video data interface from an image acquisition device and non-video data through a non-video data interface from a non-video data source, the image acquisition device and the non-video data source being collocated with a vehicle;
using first timing information received with the video data and second timing information received with the non-video data to synchronize the video data and the non-video data into synchronized vehicle data, datasets of the synchronized vehicle data having a common timeline, the common timeline indicating individual time points to which the datasets of the synchronized vehicle data respectively correspond, wherein the first timing information is generated by a video encoder of the image acquisition device;
identifying specific labels to be applied to the synchronized vehicle data, the labels indicating ground truths for the synchronized vehicle data; and
generating, by one or more computing devices collocated with the vehicle, labeled vehicular data from the synchronized vehicle data and the specific labels.

10. The media of claim 9 , wherein at least a portion of the labeled vehicular data is used for testing and evaluation of assisted driving functionality of the vehicle and wherein the assisted driving functionality includes generating predictions for driving decisions in different driving conditions.

11. The media of claim 9 , wherein the image acquisition device is at least one from a group including a mobile phone camera, an external camera attached to a device collocated in the vehicle, a LIDAR device, and combinations thereof.

12. The media of claim 9 , wherein the non-video data source represents a data source communicated through one or more from a group including: FlexRay protocols, CAN protocols, TCP/IP protocols, USB protocols, BlueRay protocols, Wi-Fi protocols, and combinations thereof.

13. The media of claim 9 , wherein the labels comprise one or more of: manual tagging labels, or automatic tagging labels.

14. The media of claim 13 , wherein the automatic tagging labels comprise one or more labels automatically generated by neural networks implemented on an end user device collocated with the vehicle.

15. The media of claim 13 , wherein the manual tagging labels comprise one or more labels determined at least in part from user input received through a user interface.

16. The media of claim 9 , wherein the one or more computing devices is at least one mobile device and wherein the at least one mobile device performs the collecting, synchronizing and identifying steps.

17. A system, comprising: one or more computing processors; one or more non-transitory computer readable media storing a program of instructions that is executable by the one or more computing processors to perform:
collecting, asynchronously, video data through a video data interface from an image acquisition device and non-video data through a non-video data interface from a non-video data source, the image acquisition device and the non-video data source being collocated with a vehicle;
using first timing information received with the video data and second timing information received with the non-video data to synchronize the video data and the non-video data into synchronized vehicle data, datasets of the synchronized vehicle data having a common timeline, the common timeline indicating individual time points to which the datasets of the synchronized vehicle data respectively correspond, wherein the first timing information is generated by a video encoder of the image acquisition device;
identifying specific labels to be applied to the synchronized vehicle data, the labels indicating ground truths for the synchronized vehicle data; and
generating, by one or more computing devices collocated with the vehicle, labeled vehicular data from the synchronized vehicle data and the specific labels.

18. The system of claim 17 , wherein at least a portion of the labeled vehicular data is used for testing and evaluation of assisted driving functionality of the vehicle and wherein the assisted driving functionality includes generating predictions for driving decisions in different driving conditions.

19. The system of claim 17 , wherein the image acquisition device is at least one from a group including a mobile phone camera, an external camera attached to a device collocated in the vehicle, a LIDAR device, and combinations thereof.

20. The system of claim 17 , wherein the non-video data source represents a data source communicated through one or more from a group including: FlexRay protocols, CAN protocols, TCP/IP protocols, USB protocols, BlueRay protocols, Wi-Fi protocols, and combinations thereof.

21. The s

Assignees

• Volkswagen Ag
• Audi Ag
• Porsche Ag

Inventors

• Tolstov Oleg
• Zhang Wenjing

Classifications

• G06F18/214

  Generating training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06F18/40

  Software arrangements specially adapted for pattern recognition, e.g. user interfaces or toolboxes therefor · CPC title

• G06F18/251 Primary

  of input or preprocessed data · CPC title

• G06F18/217

  Validation; Performance evaluation; Active pattern learning techniques · CPC title

• G06V20/70

  Labelling scene content, e.g. deriving syntactic or semantic representations · CPC title