Multi-camera vehicle vision system with image gap fill

Patent metadata

Field	Value
Publication number	US-10187590-B2
Application number	US-201615334365-A
Country	US
Kind code	B2
Filing date	Oct 26, 2016
Priority date	Oct 27, 2015
Publication date	Jan 22, 2019
Grant date	Jan 22, 2019

Abstract

A vision system of a vehicle includes an image processor and a plurality of cameras disposed at a vehicle and having respective fields of view exterior of the vehicle. The image processor provides warping or projection of a source image into a target image. The image processor scans through source images in a regular order, wherein each element of the source image is included only once in the scan order. The image processor determines the location of the source image element in the target image, with multiple source image elements existing with the same target image location. The image processor includes a set of accumulator registers, with one accumulator register selected by the location in the target image, and the image processor adds each source image element to the selected accumulator register to obtain an average of all source image elements with the same target image location.

First claim

The invention claimed is:

1. A vision system of a vehicle, said vision system comprising:
a plurality of cameras configured to be disposed at a vehicle so as to have respective fields of view exterior of the vehicle, wherein each camera captures image data, and wherein each camera comprises a two dimensional array of photosensing elements;
an image processor receiving, with the cameras disposed at the vehicle, image data captured by the cameras, the image processor processing image data captured by each of the cameras for warping or projection of a source image into a target image for display at a display device that is viewable by a driver of the vehicle;
wherein the display device comprises a two dimensional array of display pixels, the display device operable to display target images derived from image data captured by the cameras;
wherein said vision system provides images for display by the display device that are derived via backward projection, tracing rays from a display plane pixel grid of the display device backwards to one or more source camera pixel grids of one or more of the cameras using warping and unwarping schemes of the one or more cameras and virtual views that are to be generated for display of images by the display device;
wherein the image processor scans through image data of source images in a scan order, and wherein each source image element of the source image is included only once in the scan order, and wherein the image processor determines the location of a source image element in the target image, and wherein multiple source image elements exist with the same target image location; and
wherein the image processor comprises a set of accumulator registers, and wherein one accumulator register is selected by the location of the source image element in the target image, and wherein the image processor adds each source image element to the selected accumulator register to obtain an average of all source image elements within the same target image location.

2. The vision system of claim 1 , wherein one accumulator register exists per horizontal or vertical element of the target image, and wherein the accumulator register is selected by the horizontal or vertical position of the source image element in the target image.

3. The vision system of claim 1 , wherein gaps in the target image exist where no source image element is located, and wherein the gaps are between target image elements where source image elements are located, and wherein the image processor detects the gaps while scanning through the source image, and wherein the image processor comprises a memory where a list of detected gaps is maintained, and wherein the image processor uses the list of detected gap in an independent processing step to fill the gaps.

4. The vision system of claim 3 , wherein the gap fill values are calculated from one or more values of the surrounding valid image elements.

5. The vision system of claim 4 , wherein the gap is expressed as a polygon, where the polygon corners are defined by the location of surrounding valid image elements, and wherein the gap is processed line by line or column by column of target image elements.

6. The vision system of claim 5 , wherein in a first step, the image processor calculates the two intersection points between the current line or column and the edges of the gap polygon, and wherein in a second step, the image processor calculates the values of each intersection point from the values of the two corner points of the crossed polygon edge, and wherein in a third step, the image processor calculates the values of the target image elements between the intersections from the values of the intersection points.

7. The vision system of claim 3 , wherein multiple source images are warped and transformed into the same target image, where the source images represent the same scene from different views.

8. The vision system of claim 7 , wherein in a first step, all source images are processed independently into partial target images, and wherein in a second step, the gaps in the target image are filled, and wherein in a third step, all partial target images are blended into the final target image.

9. The vision system of claim 8 , wherein the first step is executed on independent entities of the image processor and the second step is executed on a common entity of the image processor.

10. The vision system of claim 3 , wherein a source image is partially or completely visible multiple times in the target image, and wherein in a first step, all views of the source image are processed independently into partial target image, and wherein in a second step, the gaps in the target image are filled, and wherein in a third step, all partial target images are blended into the final target image.

11. The vision system of claim 10 , wherein the first step is executed on independent entities of the image processor and the second step is executed on a common entity of the image processor.

12. A vision system of a vehicle, said vision system comprising:
a plurality of cameras configured to be disposed at a vehicle so as to have respective fields of view exterior of the vehicle, wherein each camera captures image data, and wherein each camera comprises a two dimensional array of photosensing elements;
an image processor receiving, with the cameras disposed at the vehicle, image data captured by the cameras, the image processor processing image data captured by each of the cameras for warping or projection of a source image into a target image for display at a display device that is viewable by a driver of the vehicle;
wherein the display device comprises a two dimensional array of display pixels, the display device operable to display target images derived from image data captured by the cameras;
wherein said vision system provides images for display by the display device that are derived via backward projection, tracing rays from a display plane pixel grid of the display device backwards to one or more source camera pixel grids of one or more of the cameras using warping and unwarping schemes of the one or more cameras and virtual views that are to be generated for display of images by the display device;
wherein the image processor scans through image data of source images in a scan order, and wherein each source image element of the source image is included only once in the scan order, and wherein the image processor determines the location of a source image element in the target image, and wherein multiple source image elements exist with the same target image location;
wherein the image processor comprises a set of accumulator registers, and wherein one accumulator register is selected by the location of the source image element in the target image, and wherein the image processor adds each source image element to the selected accumulator register to obtain an average of all source image elements within the same target image location;
wherein one accumulator register exists per horizontal or vertical element of the target image, and wherein the accumulator register is selected by the horizontal or vertical position of the source image element in the target image; and
wherein gaps in the target image exist where no source image element is located, and wherein the gaps are between target image elements where source image elements are located, and wherein the image processor detects the gaps while scanning through the source image, and wherein the image processor comprises a memory where a list of detected gaps is maintained, and wherein the image processor uses the list of detected gap in an independent processing step to fill the gaps.

Assignees

• Magna Electronics Inc

Inventors

• Fluegel Sebastian

Classifications

• H04N23/90

  Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums · CPC title

• H04N23/63

  by using electronic viewfinders · CPC title

• H04N23/698

  for achieving an enlarged field of view, e.g. panoramic image capture · CPC title

• H04N5/265 Primary

  Mixing · CPC title

• H04N7/181

  for receiving images from a plurality of remote sources · CPC title