Imaging surface modeling for camera modeling and virtual view synthesis

Patent metadata

Field	Value
Publication number	US-9858639-B2
Application number	US-201514944409-A
Country	US
Kind code	B2
Filing date	Nov 18, 2015
Priority date	Oct 11, 2012
Publication date	Jan 2, 2018
Grant date	Jan 2, 2018

Abstract

A method of displaying a captured image on a display device. A real image is captured by an image capture device. The image capture device uses a field-of-view lens that distorts the real image. A camera model is applied to the captured real image. The camera model maps objects in the captured real image to an image sensor plane of the image capture device to generate a virtual image. The image sensor plane is reconfigurable to virtually alter a shape of the image sensor plane to a non-planar surface. The virtual image formed on the non-planar image surface of the image sensor is projected to the display device.

First claim

What is claimed is:

1. A method of displaying a captured image on a display device comprising the steps of:
capturing a real image by an image capture device, the image capture device using a field-of-view lens that distorts the real image;
applying a camera model to the captured real image, the camera model mapping objects in the captured real image to an image sensor plane of the image capture device to generate a virtual image, the image sensor plane reconfigured to virtually alter a shape of the image sensor plane to a non-planar surface; and
projecting the virtual image formed on the non-planar image surface of the image sensor to the display device;
wherein generating the virtual image comprises the steps of:
providing a pre-calibrated real camera model by the processor, the real camera model representative of the vision-based imaging device capturing the scene;
determining real incident ray angles of each pixel in the captured image based on the pre-calibrated real camera model;
identifying an arbitrary shape of the non-planar imaging surface
identifying a pose of the virtual camera model;
determining virtual incident ray angles of each pixel in the virtual image based on the virtual image model and the non-planar imaging surface;
mapping a virtual incident ray to a correlated real incident ray of the real image capture device, wherein rotational compensation is applied to the virtual incident ray angles for correlating the virtual incident ray and the real incident ray if a pose of the virtual camera model is different from a pose of the of the real image capture device;
mapping pixels in the virtual image corresponding to coordinates on the non-planar virtual imaging surface to correlated pixels on the real captured image as a function of the correlation mapping between the real incident ray and the virtual incident ray, and wherein a horizontal projection of a virtual incident angle θ virt is represented by the angle α, wherein angle α is represented by the following formula:

u
virt

-

u
0

f
u

=
α

where u virt is a virtual image point u-axis coordinate, f u is a u axis focal length of the virtual camera, and u 0 is an image center u-axis coordinate.

2. The method of claim 1 wherein applying a camera model to the captured real image includes applying the camera model without radial distortion correction.

3. The method of claim 1 wherein a respective real image pixel on the real image is represented by coordinates u ureal and v real for identifying where the real incident ray is imaged on a real image surface, wherein the real incident ray is represented by the angles (θ real , φ real ), wherein θ real is an angle between the real incident ray and an optical axis of a real image capture device, and φ real is the angle between a real image capture device x axis and a projection of the real incident ray on a real image capture device x-y plane.

4. The method of claim 3 wherein the real incident ray angle is determined based on a predefined and calibrated real camera model.

5. The method of claim 4 wherein the real camera model includes a fisheye camera model, wherein the fisheye camera model represents a fisheye lens having substantial radial distortion for generating an ultra-wide angle field-of-view scene on a flat imaging plane with limited sensor size, wherein the correlation between the real incident ray and a real image pixel on the real image sensor from the real image capture device is represented as follows:

Incident
⁢

⁢
ray

→

[

θ
⁢

:

⁢

⁢
angle
⁢

⁢
between
⁢

⁢
incident
⁢

⁢
ray

and
⁢

⁢
optical
⁢

⁢
axis

φ
⁢

:

⁢

⁢
angle
⁢

⁢
between
⁢

⁢

x

c
⁢

⁢
1

⁢

Assignees

• Gm Global Tech Operations Llc

Inventors

• Zhang Wende
• Wang Jinsong
• Litkouhi Bakhtiar B
• Lybecker Kent S
• Frakes Ryan M

Classifications

• H04N23/698

  for achieving an enlarged field of view, e.g. panoramic image capture · CPC title

• B60R2300/306

  using a re-scaling of images · CPC title

• B60R2300/802

  for monitoring and displaying vehicle exterior blind spot views · CPC title

• G06T7/75

  involving models · CPC title

• G06T2207/30244

  Camera pose · CPC title