Enhanced perspective view generation in a front curb viewing system

Patent metadata

Field	Value
Publication number	US-9834143-B2
Application number	US-201414280489-A
Country	US
Kind code	B2
Filing date	May 16, 2014
Priority date	May 23, 2013
Publication date	Dec 5, 2017
Grant date	Dec 5, 2017

Abstract

A system and method for creating an enhanced perspective view of an area in front of a vehicle, using images from left-front and right-front cameras. The enhanced perspective view removes the distortion and exaggerated perspective effects which are inherent in wide-angle lens images. The enhanced perspective view uses a camera model including a virtual image surface and other processing techniques which provide corrections for two types of problems which are typically present in de-warped perspective images—including a stretching effect at the peripheral area of a wide-angle image de-warped by rectilinear projection, and double image of objects in an area where left-front and right-front camera images overlap.

First claim

What is claimed is:

1. A system for providing enhanced perspective viewing of an area in front of a vehicle, said system comprising:
a first camera mounted at the left front of the vehicle;
a second camera mounted at the right front of the vehicle;
an image acquisition module receiving raw images from the cameras, where the raw images from the cameras have a generally forward view perspective;
a processor including a memory module, said processor being configured to provide a virtual perspective view of the area in front of the vehicle, where the virtual perspective view is synthesized from the raw images from the cameras, the virtual perspective view uses a camera image surface model which corrects for artificial magnification and stretching effects of the image, and where the camera image surface model is a model of a surface onto which pixels from the raw images are projected, and the surface is in a shape of a horizontal quarter-cylinder for a below-horizon area of the raw images and is a flat planar shape for an above-horizon area of the raw images, and the virtual perspective view includes corrections in an overlap region of the raw images from the first camera and the second camera; and
a display unit in a cockpit area of the vehicle for displaying the virtual perspective view from the processor for viewing by a driver of the vehicle.

2. The system of claim 1 wherein the camera image surface model is configured such that vertical objects in front of the vehicle appear as vertical in the virtual perspective view.

3. The system of claim 1 wherein the processor uses temporal filling techniques to correct the overlap region of the raw images, where the temporal filling techniques use actual image data from previous time samples, along with vehicle motion data, to produce virtual image data for a current time in the overlap region of the raw images.

4. The system of claim 1 wherein the processor uses video morphing techniques to correct the overlap region of the raw images, where the video morphing techniques identify feature points in the overlap region of the raw images and transform the raw images so that the feature points co-align in the virtual perspective view.

5. The system of claim 1 wherein the processor uses structure-from-motion techniques to correct the overlap region of the raw images, where the structure-from-motion techniques use a sequence of the raw images and object data from a vehicle radar or lidar system to build three-dimensional models of objects in front of the vehicle, and the three-dimensional models are used to eliminate double-imaging and blank spots in the overlap region of the raw images.

6. The system of claim 1 wherein the virtual perspective view is displayed in a first window on the display unit and a bird's-eye view is displayed in a second window on the display unit, where the first window is larger than the second window, and the bird's-eye view includes a depiction of the vehicle in the center with a forward-view image in front of the vehicle depiction and a rear-view image behind the vehicle depiction.

7. The system of claim 1 wherein the virtual perspective view includes an indication of whether a curb in front of the vehicle will make contact with a low-hanging body component of the vehicle.

8. A method for providing enhanced perspective viewing of an area in front of a vehicle, said method comprising:
providing images from front-mounted cameras on the vehicle, where the images have a generally forward view perspective;
synthesizing a virtual perspective view from the images from the cameras;
performing, using a microprocessor, a de-warping calculation on the virtual perspective view to produce a de-warped virtual perspective view;
enhancing the de-warped virtual perspective view by applying a virtual camera image surface model which corrects for artificial magnification and stretching effects of image de-warping, creating an enhanced virtual perspective view, where the virtual camera image surface model is a model of a surface onto which pixels from the images are projected, and the surface is in a shape of a horizontal quarter-cylinder for a below-horizon area of the images and is a flat planar shape for an above-horizon area of the images;
correcting, in the enhanced virtual perspective view, double-image and blank region discrepancies which exist in an overlap area of the images from the front-mounted cameras; and
displaying the enhanced virtual perspective view on a display device for viewing by a driver.

9. The method of claim 8 wherein the virtual camera image surface model is configured such that vertical objects in front of the vehicle appear as vertical in the enhanced virtual perspective view.

10. The method of claim 8 wherein correcting double-image and blank region discrepancies includes using temporal filling techniques to correct the overlap area of the images, where the temporal filling techniques use actual image data from previous time samples, along with vehicle motion data, to produce virtual image data for a current time in the overlap area.

11. The method of claim 8 wherein correcting double-image and blank region discrepancies includes using video morphing techniques to correct the overlap area of the images, where the video morphing techniques identify feature points in the overlap area and transform the raw images so that the feature points co-align in the enhanced virtual perspective view.

12. The method of claim 8 wherein correcting double-image and blank region discrepancies includes using structure-from-motion techniques to correct the overlap area of the images, where the structure-from-motion techniques use a sequence of the images and object data from a vehicle radar or lidar system to build three-dimensional models of objects in front of the vehicle, and the three-dimensional models are used to eliminate double-imaging and blank spots in the enhanced virtual perspective view.

13. The method of claim 8 wherein the enhanced virtual perspective view is displayed in a first window on the display device and a bird's-eye view is displayed in a second window on the display device, where the first window is larger than the second window, and the bird's-eye view includes a depiction of the vehicle in the center with a forward-view image in front of the vehicle depiction and a rear-view image behind the vehicle depiction.

14. The method of claim 8 wherein the enhanced virtual perspective view includes an indication of whether a curb in front of the vehicle will make contact with a low-hanging body component of the vehicle.

15. A method for providing enhanced perspective viewing of an area in front of a vehicle, said method comprising:
providing images from front-mounted cameras on the vehicle, where the images have a generally forward view perspective;
synthesizing a virtual perspective view from the images from the cameras;
performing, using a microprocessor, a de-warping calculation on the virtual perspective view to produce a de-warped virtual perspective view;
enhancing the de-warped virtual perspective view by applying a virtual camera image surface model which corrects for artificial magnification and stretching effects of image de-warping, where the virtual camera image surface model has a quarter-cylindrical shape applied to a below-horizon area and a flat planar shape applied to an above-horizon area, creating an enhanced virtual perspective view;
correcting, in the enhanced virtual perspective view, double-image and blank region discrepancies in an overlap area of the images from the front-mounted cameras, including image render

Assignees

• Gm Global Tech Operations Llc

Inventors

• Zhang Wende
• Wang Jinsong
• Lybecker Kent S
• Piasecki Jeffrey S
• Litkouhi Bakhtiar Brian
• Frakes Ryan M

Classifications

• H04N7/183

  for receiving images from a single remote source · CPC title

• B60R1/002 Primary

  specially adapted for covering the peripheral part of the vehicle, e.g. for viewing tyres, bumpers or the like (integrated in the windows B60R1/001; combined with rear-view mirrors B60R1/08) · CPC title

• B60R2011/004

  outside the vehicle · CPC title

• B60R1/00

  Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles · CPC title

• G06T5/006

  Physics · mapped topic