Three Dimensional Rendering for Surround View Using Predetermined Viewpoint Lookup Tables

Patent metadata

Field	Value
Publication number	US-2017195564-A1
Application number	US-201615298214-A
Country	US
Kind code	A1
Filing date	Oct 19, 2016
Priority date	Jan 6, 2016
Publication date	Jul 6, 2017
Grant date	—

Abstract

A method for displaying a surround view on a single display screen is disclosed. A plurality of image frames for a particular time may be received from a corresponding plurality of cameras. A viewpoint warp map corresponding to a predetermined first virtual viewpoint may be selected, wherein the viewpoint warp map defines a source pixel location in the plurality of image frames for each output pixel location in the display screen. The warp map was predetermined offline and stored for later use. An output image is synthesized for the display screen by selecting pixel data for each pixel of the output image from the plurality of image frames in accordance with the viewpoint warp map. The synthesized image is then displayed on a display screen.

First claim

What is claimed is:

1 . A method for displaying a surround view on a single display screen, the method comprising:
receiving a plurality of image frames for a particular time from a corresponding plurality of cameras; selecting a first warp map corresponding to a predetermined first virtual viewpoint, wherein the warp map defines a source pixel location in the plurality of image frames for each output pixel location in the display screen, in which the first warp map was predetermined and stored for later use; synthesizing an output image for the display screen by selecting pixel data for each pixel of the output image from the plurality of image frames in accordance with the first warp map; and displaying the output image on the display screen.

2 . The method of claim 1 , further including selecting different ones of a plurality of predetermined warp maps corresponding to a plurality of different predetermined virtual viewpoints in a sequential manner to produce a video sequence on the display screen.

3 . The method of claim 1 , in which the plurality of cameras are mounted on a vehicle and in which the first warp map was produced using a plurality of test cameras in a test bed.

4 . The method of claim 3 , in which the first warp map was combined with calibration parameters to account for differences in location between the plurality of cameras and corresponding ones of the plurality of test cameras during an offline setup process.

5 . The method of claim 3 , further including combining a first raw warp map with calibration parameters to account for differences in location between the plurality of cameras and corresponding ones of the plurality of test cameras using a processing module mounted on the vehicle to form the first warp map.

6 . The method of claim 5 , in which the first warp map is stored in nonvolatile memory located on the vehicle.

7 . The method of claim 1 , in which the plurality of cameras are mounted on a vehicle, further including selecting a first vehicle image and overlying it on the output image, in which the first vehicle image was previously extracted from a model vehicle corresponding to the first viewpoint and stored for later use.

8 . The method of claim 1 , in which synthesizing the output image renders a separate virtual view corresponding to each of the plurality of cameras by warping each image frame to correct lens distortion using the first warp map; and further including stitching and blending two or more of the separate virtual views to form the output image.

9 . A method for displaying a surround view on a single display screen, the method comprising:
defining a view surface representing a three dimensional (3D) view; positioning a plurality of test cameras in positions to capture images from the view surface; generating a first warp map for a first virtual view point, which includes casting a ray from a two dimensional (2D) pixel location in an output frame corresponding to the first virtual view point to intersect with the view surface and mapping that ray to a corresponding pixel location in one of the plurality of test cameras, and repeating the step of casting a ray and mapping that ray for each 2D pixel location of the output frame; and storing the first warp map for later use.

10 . The method of claim 9 , further including generating a plurality of warp maps corresponding to a plurality of virtual view points and storing the plurality of warp maps for later use.

11 . The method of claim 10 , in which the plurality of cameras are positioned for mounting on a vehicle, further including:
extracting a plurality of vehicle images of a model vehicle from the plurality of virtual viewpoints; and storing the plurality of vehicle images for later use with corresponding ones of the plurality of warp maps.

12 . The method of claim 9 , further including:
storing the plurality of warp maps in a processing system on a first vehicle; determining a set of camera calibration parameters for a plurality of live cameras mounted on the first vehicle to account for differences in position between the plurality of live cameras and corresponding ones of the plurality of test cameras; updating the plurality of warp maps using the set of camera calibration parameters to form a plurality of calibrated warp maps; and storing the set of calibrated warp maps in the processing system of the vehicle for later use.

13 . A 3D surround view system comprising:
a vehicle; a plurality of cameras mounted on the vehicle; a display screen mounted on the vehicle for viewing by a driver of the vehicle; an image processing system coupled to receive a plurality of image frames for a particular time from the plurality of cameras, having an output coupled to provide an output image to the display screen, in which the image processing system is configured to select a first warp map corresponding to a predetermined first virtual viewpoint, wherein the warp map defines a source pixel location in the plurality of image frames for each output pixel location in the display screen, in which the first warp map was predetermined and stored for later use; and to synthesize the output image for the display screen by warping each image frame to correct lens distortion using the first warp map to form a separate virtual view corresponding to each image frame.

14 . The system of claim 13 , in which the image processing system is further configured to stitch and blend two or more of the separate virtual views to form the output image.

15 . The system of claim 13 , in which the image processing system is further configured to select different ones of a plurality of predetermined warp maps corresponding to a plurality of different predetermined virtual viewpoints in a sequential manner to produce a video sequence on the display screen.

16 . The system of claim 13 , in which the first warp map was produced using a plurality of test cameras in a test bed.

17 . The system of claim 16 , in which the first warp map was combined with calibration parameters to account for differences in location between the plurality of cameras and corresponding ones of the plurality of test cameras during an offline setup process.

18 . The system of claim 16 , in which the image processing system is further configured to combine a first raw warp map with calibration parameters to account for differences in location between the plurality of cameras and corresponding ones of the plurality of test cameras using a processing module mounted on the vehicle to form the first warp map.

19 . The system of claim 18 , in which the first warp map is stored in nonvolatile memory located on the vehicle.

20 . The system of claim 13 , in which the image processing system is further configured to select a first vehicle image and overly it on the output image, in which the first vehicle image was previously extracted from a model vehicle corresponding to the first viewpoint and stored for later use.

Assignees

• Texas Instruments Inc

Inventors

• Appia Vikram Vijayanbabu
• Shivalingappa Sujith
• Jadav Brijesh Rameshbhai
• Hariyani Hemant
• Dabral Shashank
• Mangla Mayank

Classifications

• H04N23/90

  Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums · CPC title

• H04N25/61

  the noise originating only from the lens unit, e.g. flare, shading, vignetting or "cos4" · CPC title

• H04N23/698 Primary

  for achieving an enlarged field of view, e.g. panoramic image capture · CPC title

• B60R2300/105

  using multiple cameras · CPC title

• B60R2300/802

  for monitoring and displaying vehicle exterior blind spot views · CPC title