Virtual viewpoint synthesis method and system

Patent metadata

Field	Value
Publication number	US-9838663-B2
Application number	US-201615009854-A
Country	US
Kind code	B2
Filing date	Jan 29, 2016
Priority date	Jul 29, 2013
Publication date	Dec 5, 2017
Grant date	Dec 5, 2017

Abstract

A virtual viewpoint synthesis method and system, including: establishing a left viewpoint virtual view and a right viewpoint virtual view; searching for a candidate pixel in a reference view, and marking a pixel block in which the candidate pixel is not found as a hole point; ranking the found candidate pixels according to depth, and successively calculating a foreground coefficient and a background coefficient for performing weighted summation; enlarging the hole-point regions of the left viewpoint virtual view and/or the right viewpoint virtual view in the direction of the background to remove a ghost pixel; performing viewpoint synthesis on the left viewpoint virtual view and the right viewpoint virtual view; and filling the hole-points of a composite image.

First claim

The invention claimed is:

1. A virtual viewpoint synthesis method, comprising:
establishing a left viewpoint virtual view and a right viewpoint virtual view and obtaining (u, v) to (u+1, v) pixel blocks for a pixel with a coordinate of (u, v) in a virtual viewpoint, that is, I(u)=[u, u+1);
searching for a candidate pixel in a left viewpoint reference view using the left viewpoint virtual view, searching for a candidate pixel in a right viewpoint reference view using the right viewpoint virtual view, marking a pixel block in which a candidate pixel isn't found in the left viewpoint reference view and/or the right viewpoint reference view as a hole point;
ranking the found candidate pixels according to depth, successively calculating a foreground coefficient and a background coefficient, and performing a weighted summation according to the foreground coefficient and the background coefficient, and, not including the pixels in the weighted summation if difference between depth value of current pixel and depth value of an initial pixel in the weighted summation exceeds a first predetermined threshold value;
enlarging hole-point regions of the left viewpoint virtual view and/or the right viewpoint virtual view in the direction of background to remove a ghost pixel;
performing viewpoint synthesis on the left viewpoint virtual view and the right viewpoint virtual view; and
filling the hole-points of a composite image.

2. The method of claim 1 , comprising:
searching for a candidate pixel in a reference view and setting a first adjustable parameter; adding 1 to the first adjustable parameter for a new search if the pixel block in which a candidate pixel isn't found when the first adjustable parameter is a preset initial value; and marking the pixel block as a hole point if the pixel block in which a candidate pixel is still not found.

3. The method of claim 1 , wherein:
the foreground coefficient is calculated by w j f =|I j (u r , t)∩I j (u)|;
the background coefficient is calculated by w j b =|I(u)∩I j (u r , t)|−w j f ;
wherein, I j (u r , t)=[u r −t, u r +1+t) represents a pixel block in a reference viewpoint, t represents a second adjustable parameter, and I j (u) represents a specific gravity which isn't covered by previous pixels in a pixel block of a virtual viewpoint;
a formula for performing the weighted summation according to the foreground coefficient and the background coefficient is as below:

⁢

Y
⁢

(
u
)

=

∑

j
=
1

⁢



C
⁡

(
u
)



⁢

(

w
j
f

+

α
j

×

w
j
b

)

⁢

Y
⁡

(
j
)

∑

j
=
1

⁢



C
⁡

(
u
)



⁢

(

w
j
f

+

α
j

×

w
j
b

)

,

wherein, α j controls pixel weights in a synthesis process according to depth values of pixels.

4. The method of claim 1 , wherein: the step of enlarging the hole-point regions of the left viewpoint virtual view and/or the right viewpoint virtual view in the direction of the background to remove a ghost pixel, comprising:
scanning the left viewpoint and/or right viewpoint virtual views line by line, recording an initial position of the hole point when scanning a hole point and then continuing to scan the views towards the right to find right boundary of the hole-point region; then, choosing a predetermined amount of pixels from the left and right boundaries outward in succession separately, and calculating average depth values of pixels which are respectively recorded as d_left and d_right; calculating a difference value between d_left and d_right; and then comparing the difference value between d_left and d_right to a second predetermined threshold value to determine enlarging direction of hole-point regions, wherein:
if d_left−d_right is greater than the second predetermined threshold value, the hole-point region is enlarged towards the left;
if d_left−d_right is less than the second predetermined threshold value, the hole-point region is enlarged towards the right; and
if d_left−d_right equals to the second predet

Assignees

• Univ Peking Shenzhen Graduate School

Inventors

• Li Chenxia
• Wang Ronggang
• Gao Wen

Classifications

• H04N13/0018

  Electricity · mapped topic

• H04N2013/0081

  Depth or disparity estimation from stereoscopic image signals · CPC title

• H04N13/0011 Primary

  Electricity · mapped topic

• H04N13/128

  Adjusting depth or disparity · CPC title

• H04N13/111 Primary

  Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation · CPC title