Method and system for representation of vehicle surroundings with reduced artefacts

Patent metadata

Field	Value
Publication number	US-10943334-B2
Application number	US-201916520893-A
Country	US
Kind code	B2
Filing date	Jul 24, 2019
Priority date	Jul 24, 2018
Publication date	Mar 9, 2021
Grant date	Mar 9, 2021

Abstract

A system is configured to repair artefacts in image data when representing vehicle surroundings. This involves: detecting an artefact-comprising area of a vehicle camera for a first time point, creating a reconstructed version of the artefact-comprising area from image data of an adjacent image area, determining a sample point of the artefact-comprising area, transforming the sample point to a new location predicted for a second time point, determining whether the new location still lies within the artefact-comprising area, and if so updating the artefact-comprising area in the image for the second time point by using motion-compensated image data from the reconstructed version of the sample point from the first time point.

First claim

The invention claimed is:

1. A method of representing a vehicle surroundings in a moving vehicle, comprising steps:
with a vehicle camera of the moving vehicle, capturing a first image of the vehicle surroundings at a first time point and a second image of the vehicle surroundings at a second time point,
detecting an artefact-comprising area in the first image captured at the first time point, wherein the artefact-comprising area in the first image depicts an artefact of the vehicle camera, wherein the artefact is caused by soil or dirt present on a lens of the vehicle camera,
creating a reconstructed version of the artefact-comprising area in the first image captured at the first time point by using image data that comprises texture data of an adjacent image area that is adjacent to the artefact-comprising area in the first image captured at the first time point,
determining a sample point of the artefact-comprising area in the first image captured at the first time point,
transforming the sample point to a new location of the sample point predicted for the second time point based on a motion of the vehicle,
determining whether the new location still lies within the artefact-comprising area, and
when the new location still lies within the artefact-comprising area, updating the artefact-comprising area in the second image captured at the second time point by using reconstructed image data of the sample point in the reconstructed version of the artefact-comprising area in the first image captured at the first time point and motion-compensating the reconstructed image data to the new location for the second time point based on the motion of the vehicle.

2. The method according to claim 1 , further comprising, when the new location lies outside the artefact-comprising area, rendering the artefact-comprising area in the second image captured at the second time point by using image data of the sample point of the first image captured at the first time point.

3. The method according to claim 2 , further comprising determining, by a motion vector obtained from vehicle motion data regarding the motion of the vehicle, which particular vehicle camera of a plurality of vehicle cameras of the vehicle having different viewing areas captured the first image at the first time point.

4. The method according to claim 1 , wherein determining the sample point comprises determining a world point of the sample point in a real world space.

5. The method according to claim 1 , further comprising obtaining motion data regarding the motion of the vehicle, for performing the motion compensation, by an odometry system of the vehicle.

6. The method according to claim 4 , wherein the world point is determined based on calibration data of the vehicle camera.

7. The method according to claim 1 , further comprising iteratively repeating the step of the determining of the sample point respectively for plural sample points until all sample points of the artefact-comprising area have been determined.

8. A system for representing a vehicle surroundings in a moving vehicle, comprising:
a vehicle camera of the moving vehicle configured to capture a first image of the vehicle surroundings at a first time point and a second image of the vehicle surroundings at a second time point,
a memory configured to store the first image captured at the first time point and the second image captured at the second time point, and
a data processing unit configured to:
detect an artefact-comprising area in the first image captured at the first time point, wherein the artefact-comprising area in the first image depicts an artefact of the vehicle camera, wherein the artefact is caused by soil or dirt present on a lens of the vehicle camera,
create a reconstructed version of the artefact-comprising area in the first image captured at the first time point by using image data that comprises texture data of an adjacent image area that is adjacent to the artefact-comprising area in the first image captured at the first time point,
determine a sample point of the artefact-comprising area in the first image captured at the first time point,
transform the sample point to a new location of the sample point predicted for the second time point based on a motion of the vehicle,
determine whether the new location still lies within the artefact-comprising area, and
when the new location still lies within the artefact-comprising area, update the artefact-comprising area in the second image captured at the second time point by using reconstructed image data of the sample point in the reconstructed version of the artefact-comprising area in the first image captured at the first time point and motion-compensating the reconstructed image data to the new location for the second time point based on the motion of the vehicle.

9. The system according to claim 8 , wherein the data processing unit is further configured, when the new location lies outside the artefact-comprising area, to render the artefact-comprising area in the second image captured at the second time point by using image data of the sample point of the first image captured at the first time point.

10. The method according to claim 7 , further comprising iteratively repeating the steps of the transforming of the sample point to the new location, the determining of whether the new location lies within the artefact-comprising area, and the updating of the artefact-comprising area, respectively for all of the sample points of the artefact-comprising area.

11. The method according to claim 1 , wherein the artefact comprises a temporary static artefact of the vehicle camera caused by the soil or the dirt present on the lens of the vehicle camera.

12. The method according to claim 1 , wherein the artefact comprises a static artefact caused by the soil or the dirt present on the lens of the vehicle camera.

13. The method according to claim 1 , wherein the artefact-comprising area is an area of an imager of the vehicle camera, at which area of the imager the soil or the dirt present on the lens of the vehicle camera is imaged.

Assignees

• Continental Automotive Gmbh

Inventors

• Friebe Markus
• Srikanth Sukumar
• Prabhakar Pavannag

Classifications

• H04N23/683

  performed by a processor, e.g. controlling the readout of an image memory · CPC title

• H04N23/54

  Mounting of pick-up tubes, electronic image sensors, deviation or focusing coils · CPC title

• H04N23/90

  Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums · CPC title

• G06V20/56 Primary

  exterior to a vehicle by using sensors mounted on the vehicle · CPC title

• G06T2207/30252

  Vehicle exterior; Vicinity of vehicle · CPC title