Method, device and computer-readable storage medium with instructions for processing sensor data

Patent metadata

Field	Value
Publication number	US-11935250-B2
Application number	US-201917048701-A
Country	US
Kind code	B2
Filing date	Mar 27, 2019
Priority date	Apr 18, 2018
Publication date	Mar 19, 2024
Grant date	Mar 19, 2024

Abstract

A method, a device and a computer-readable storage medium with instructions for processing sensor data. In a first step, camera images are taken by a camera. Additionally, 3D-checkpoints are detected by at least one 3D-sensor. Optionally, at least one of the camera images can be segmented. The camera images are then merged with the 3D-checkpoints by a data fusion circuit to form data of a virtual sensor. The resulting data are finally output for further processing.

First claim

What is claimed is:

1. Method for processing sensor data, comprising:
obtaining camera images by a camera including a first camera image captured at a first time and a second image captured at a second time;
detecting 3D-checkpoints by at least one 3D-sensor at a 3D-checkpoint detection time after the first time of the first camera image and before the second time of the second camera image; and
fusing the camera images with the 3D-checkpoints to form data of a virtual sensor, wherein fusing the camera images with the 3D-checkpoints comprises:
calculating an optical flow from at least the first camera image and the second camera image;
selecting either the first camera image or the second camera image based on whether the 3D-checkpoint detection time is closer in time to the first time of the first camera image or the second time of the second camera image;
converting the selected first or second camera image to the 3D-checkpoint detection time based at least on the optical flow and a time difference between the respective first time or second time of the selected first or second camera image and the 3D-checkpoint detection time; and
projecting the 3D-checkpoints into the converted selected first or second selected camera image.

2. The method of claim 1 , wherein determining in at least one of the camera images pixels to be assigned to one of the 3D-checkpoints at an instant of time of the measurement comprises:
determining, on the basis of the optical flow and a search method, those pixels in the camera image which are to be assigned to the 3D-checkpoints at the instant of time of the measurement; and
projecting the 3D-checkpoints at the points determined in this way in the camera image.

3. The method of claim 2 , wherein from the optical flow a time to collision is determined for the pixels of the camera images and wherein a velocity vector for this 3D-checkpoint is calculated from the time to collision, the optical flow, and a distance measurement for a 3D-checkpoint.

4. The method of claim 2 , wherein the 3D-checkpoints are expanded to include attributes from at least one of the camera images.

5. The method of claim 1 , wherein from the optical flow a time to collision is determined for the pixels of the camera images and wherein a velocity vector for this 3D-checkpoint is calculated from the time to collision, the optical flow, and a distance measurement for a 3D-checkpoint.

6. The method of claim 5 , wherein the time to collision is determined from a measurement by the 3D-sensor instead of from the optical flow.

7. The method of claim 1 , wherein the 3D-checkpoints are expanded to include attributes from at least one of the camera images.

8. The method of claim 1 , wherein at least one camera image is segmented near an instant of time of measurement of the 3D-sensor.

9. The method of claim 8 , wherein segmenting besides image information also considers measurements of the 3D-sensor.

10. The method of claim 1 , wherein an algorithm for object tracking is applied to the data of the virtual sensor.

11. The method of claim 10 , wherein the algorithm for object tracking performs an accumulating sensor data fusion.

12. A non-transitory computer-readable storage medium with instructions which, when executed by a computer, cause the computer to carry out the method of claim 1 for processing sensor data.

13. Motor vehicle, wherein the motor vehicle is configured to carry out the method of claim 1 for processing sensor data.

14. A device for processing sensor data, comprising:
an input for receiving camera images from a camera and 3D-checkpoints from a 3D-sensor, wherein the camera images include a first camera image captured at a first time and a second image captured at a second time, and wherein the 3D-checkpointsare detected at a 3D-checkpoint detection time after the first time of the first camera image and before the second time of the second camera image; and
a data fusion circuit for fusing the camera images with the 3D-checkpoints to form data of a virtual sensor;
wherein the data fusion circuit is further configured to:
calculate an optical flow from at least the first camera image and the second camera image;
select either the first camera image or the second camera image based on whether the 3D-checkpoint detection time is closer in time to the first time of the first camera image or the second time of the second camera image;
convert the selected first or second camera image to the 3D-checkpoint detection time based at least on the optical flow and a time difference between the respective first time or second time of the selected first or second camera image and the 3D-checkpoint detection time; and
project the 3D-checkpoints into the converted selected first or second selected camera image.

15. A motor vehicle, wherein the motor vehicle comprises the device of claim 14 .

16. The device of claim 14 , wherein from the optical flow a time to collision is determined for the pixels of the camera images and wherein a velocity vector for this 3D-checkpoint is calculated from the time to collision, the optical flow, and a distance measurement for a 3D-checkpoint.

17. The device of claim 13 , wherein the 3D-checkpoints are expanded to include attributes from at least one of the camera images.

Assignees

• Volkswagen Ag

Inventors

• Steinmeyer Simon
• Musial Marek
• Deeg Carsten
• Bagdonat Thorsten
• Graf Thorsten

Classifications

• G06T7/269 Primary

  using gradient-based methods · CPC title

• G01S13/867

  Combination of radar systems with cameras · CPC title

• G01S13/931

  of land vehicles · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title