Method for temporal correction of multimodal data

Patent metadata

Field	Value
Publication number	US-12555368-B2
Application number	US-202318337153-A
Country	US
Kind code	B2
Filing date	Jun 19, 2023
Priority date	Jun 23, 2022
Publication date	Feb 17, 2026
Grant date	Feb 17, 2026

Abstract

A method for the chronological correction of multimodal data includes:
receiving a first data set from a reference sensor with measurements at different measurement timepoints, receiving a second data set of a second sensor with measurements at different measurement timepoints, each not exactly matching those of the reference sensor, reading the first and the second data sets by a neural network and identifying a respective plurality of feature vectors for the first and second data set at the respective measurement timepoints, merging and comparing the respective feature vectors, which refer to corresponding, not exactly matching measurement timepoints, by the neural network so that parameters of a chronological correction are identified, and identifying a chronological offset between the respective measurement timepoints of the reference sensor and the second sensor, and/or a corrected data set from the second sensor based on the measurement timepoints of the reference sensor.

First claim

The invention claimed is:

1 . A method for chronological correction of multimodal data, comprising:
receiving a first data set from a reference sensor with measurements at different respective measurement timepoints; receiving a second data set from a second sensor with measurements at different respective measurement timepoints, each of the different respective measurement timepoints not exactly matching those of the reference sensor; reading the first and the second data sets by a neural network and identifying a respective plurality of feature vectors for the first and second data sets at the different respective measurement timepoints; merging and comparing the respective pluralities of feature vectors by the neural network so that parameters of the chronological correction are identified; and identifying a chronological offset between the different respective measurement timepoints of the reference sensor and the second sensor, and/or a corrected data set from the second sensor based on the different respective measurement timepoints of the reference sensor.

2 . The method according to claim 1 , wherein the reference and second sensors use a same or different measurement principles, in particular radar, lidar, ultrasonic, and/or camera sensors.

3 . The method according to claim 1 , wherein measured values from the reference sensor as well as the measured values from the second sensor are each projected into a selected coordinate system, and the respective pluralities of feature vectors are extracted based on a result of these projections by the neural network.

4 . The method according to claim 3 , wherein the selected coordinate system is a 2-dimensional or 3-dimensional Cartesian coordinate system.

5 . The method according to claim 1 , further comprising:
receiving a third data set from a third sensor with measurements at different respective measurement timepoints not exactly matching those of the reference sensor or those from the second sensor; and identifying a chronological offset between the respective measurement timepoints of the reference sensor and the third sensor using a separate neural network.

6 . The method according to claim 1 , further comprising:
receiving a third data set from a third sensor with measurements at different respective measurement timepoints not exactly matching those of the reference sensor or those from the second sensor; and identifying a chronological offset between the different respective measurement timepoints of the reference sensor and the third sensor using the neural network.

7 . The method according to claim 1 , wherein the data sets each relate to environmental detection of a vehicle environment and/or a traffic situation by a system having corresponding sensors installed in a vehicle for assistance-based and/or autonomous driving, or which are part of a traffic monitoring system.

8 . The method according to claim 1 , further comprising:
identifying a control signal based on analysis of the data sets related to a common reference timepoint, the control signal being configured to trigger, as an action in a vehicle, initiation of braking, control of acceleration, control of a steering system for initiating cornering, control of lighting, control of a hazard light system, and/or control of windscreen wipers.

9 . A method for training a neural network for use in the method according to claim 1 , comprising:
providing training examples containing at least a first training data set from the reference sensor and a second training data set from the second sensor; processing the first and second training data sets using the method according to claim 1 into a training value for the chronological offset and/or into a training corrected data set from the second sensor; evaluating the value for the training chronological offset and/or the training corrected data set using a predetermined cost function in terms of its quality; and optimizing parameters and weights of the neural network to improve an assessment obtained by the cost function with further processing of the training examples.

10 . The method according to claim 9 , wherein the cost function measures a plausibility of or conformance with the training chronological offset or the training corrected data set with a target output previously known for the respective training example.

11 . The method according to claim 9 , wherein the cost function measures fulfillment of a similarity condition and/or consistency condition between the training corrected data set and the first training data set.

12 . The method according to claim 9 , wherein the cost function measures a quality of a processing product identified by downstream processing of the training corrected data set and/or by downstream processing of the training second data set with additional use of the training chronological offset identified.

13 . The method according to claim 9 , wherein the different respective measurement timepoints of both the reference sensor and the second sensor are precisely known, the method further comprising:
identifying a speed of a platform supporting both sensors as well as other potential speeds of objects in a field of view of the sensors; identifying, based on the speeds identified and the chronological offset between the different respective measurement timepoints of the sensors, a comparison data set of the second sensor related to the different respective measurement timepoints of the reference sensor; comparing the comparison data set with the training corrected data set identified by the neural network using a cost function; and adjusting, in view of minimizing the cost function, the parameters and weights of the neural network.

14 . The method according to claim 9 , wherein the cost function is based on metrics quantifying a similarity of point clouds.

15 . The method according to claim 9 , wherein the training chronological offset identified by the neural network is fed into an algorithm to be trained for environmental detection and the cost function, which also characterizes the quality of a completed environmental detection to determine the quality of the training chronological offset identified.

16 . A non-transitory machine-readable storage medium that stores machine readable instructions that, when executed on one or multiple computers, prompt the one or multiple computers to perform the method according to claim 1 .

17 . One or multiple computers equipped with the non-transitory machine-readable storage medium according to claim 16 .

Assignees

• Bosch Gmbh Robert

Inventors

• Glaeser Claudius
• Timm Fabian
• Drews Florian
• Ulrich Michael
• Faion Florian
• Rosenbaum Lars

Classifications

• G06V10/751

  Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching · CPC title

• G06V10/80

  Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level (multimodal speaker identification or verification G10L17/10) · CPC title

• G06V10/776

  Validation; Performance evaluation · CPC title

• G06F2218/16

  by matching signal segments · CPC title

• G06V10/75

  Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries · CPC title