Apparatus and method for generating new viewpoint image

Patent metadata

Field	Value
Publication number	US-9460545-B2
Application number	US-201213688316-A
Country	US
Kind code	B2
Filing date	Nov 29, 2012
Priority date	Nov 30, 2011
Publication date	Oct 4, 2016
Grant date	Oct 4, 2016

Abstract

An apparatus for generating a new viewpoint image includes: a new viewpoint image generation unit that shifts the foreground image region in the input image, based on a depth map to generate a new viewpoint image including a missing pixel region; a representative background pixel selection unit that selects background pixels located a predetermined number of pixels away from edge pixels, as representative background pixels; a texture direction map calculation unit that calculates a texture direction of each of the selected representative background pixels, to calculate a texture direction map indicating texture directions, each corresponding to one of missing pixels, based on locations of the representative background pixels and locations of the missing pixels included in the missing pixel region; and a hole-filling processing unit that fills the missing pixel region with the obtained background pixels in accordance with the texture direction map.

First claim

The invention claimed is:

1. An apparatus for generating a new viewpoint image, which generates, from an input image, a new viewpoint image that is an image of the input image viewed from a new viewpoint, the input image including a foreground image region that is an image region of an object and a background image region other than the foreground image region, the apparatus comprising:
a generation unit configured to shift the foreground image region in the input image, based on a depth map corresponding to the input image to generate a new viewpoint image including a missing pixel region, according to a depth image based rendering (DIBR) technique;
a selection unit configured to select background pixels located a predetermined number of pixels away from edge pixels, from among background pixels included in the background image region, as representative background pixels, the edge pixels indicating a boundary between the foreground image region and the background image region in the input image;
a texture direction map calculation unit configured to calculate a texture direction of each of the selected representative background pixels, the texture direction being an edge direction of texture in the background image region for each of the representative background pixels, and to calculate, for all of missing pixels, a texture direction map indicating texture directions each corresponding to one of the missing pixels, based on locations of the representative background pixels and locations of the missing pixels included in the missing pixel region; and
a hole-filling processing unit configured to entirely fill, at one time, the missing pixel region with the background pixels without using the pixels of the foreground image region, while considering continuity at a boundary with the background image region and without considering discontinuity at a boundary with the foreground image region, in accordance with the texture direction map indicating the texture directions of the missing pixels.

2. The apparatus for generating a new viewpoint image according to claim 1 ,
wherein the texture direction map calculation unit includes:
a texture direction calculation unit configured to calculate texture directions of the representative background pixels; and
a map calculation unit configured to calculate the texture direction map by weighting the texture directions of the representative background pixels based on the locations of the representative background pixels and the locations of the missing pixels and calculating the texture directions, each corresponding to one of the missing pixels.

3. The apparatus for generating a new viewpoint image according to claim 2 ,
wherein the map calculation unit is configured to calculate the texture directions, each corresponding to one of the missing pixels, by (i) calculating a direction in which weighted texture directions of the representative background pixels are summed, as one of preliminary texture directions of the missing pixels, in accordance with distances between one of the missing pixels and the representative background pixels and directions of the representative background pixels with respect to one of the missing pixels and (ii) averaging each of the calculated preliminary texture directions of the missing pixels.

4. The apparatus for generating a new viewpoint image according to claim 2 , further comprising
a reliability value calculation unit configured to calculate reliability values for the calculated texture directions of the representative background pixels,
wherein the map calculation unit is configured to calculate the texture directions, each corresponding to one of the missing pixels, by weighting a texture direction of at least one first representative background pixel, based on a location of the at least one first representative background pixel and locations of the missing pixels, each of the first representative background pixels being a pixel for which a reliability value greater than a predetermined threshold is calculated by the reliability value calculation unit, among the representative background pixels.

5. The apparatus for generating a new viewpoint image according to claim 4 ,
wherein the reliability value calculation unit is configured to calculate a reliability value for a texture direction of a representative background pixel to be calculated, based on (i) a distance from a calculated representative background pixel to the representative background pixel to be calculated and (ii) an orientation of a vector of the calculated representative background pixel with respect to the representative background pixel to be calculated.

6. The apparatus for generating a new viewpoint image according to claim 2 ,
wherein the texture direction calculation unit is configured to trace edges of texture of the background image region from the selected representative background pixels as starting points to calculate the texture directions of the selected representative background pixels.

7. The apparatus for generating a new viewpoint image according to claim 1 , further comprising
an edge strength calculation unit configured to calculate edge strength for each of the selected representative background pixels,
wherein when the edge strength of the representative background pixel is greater than a threshold, the texture direction map calculation unit is configured to calculate a texture direction of the representative background pixel.

8. The apparatus for generating a new viewpoint image according to claim 1 , further comprising
an edge detection unit configured to detect, based on the depth map, edge pixels indicating a boundary between the foreground image region and the background image region in the input image,
wherein the selection unit is configured to select, as the representative background pixels, background pixels located a predetermined number of pixels away from the edge pixels in a predetermined direction, the edge pixels being detected by the edge detection unit.

9. The apparatus for generating a new viewpoint image according to claim 8 ,
wherein the selection unit is configured to, for each of the edge pixels detected by the edge detection unit,
calculate an average value of horizontal differential coefficients and an average value of vertical differential coefficients for the edge pixel,
determine an orthogonal direction of the edge pixel as the predetermined direction, using the average value of the horizontal differential coefficients and the average value of the vertical differential coefficients, and
select, as one of the representative background pixels, a background pixel located a predetermined number of pixels away from the edge pixel in the determined orthogonal direction.

10. The apparatus for generating a new viewpoint image according to claim 1 ,
wherein the generation unit is configured to generate a new viewpoint image including the missing pixel region by (i) shifting the foreground image region in the input image, based on a depth value in the depth map, a distance between a right eye and a left eye, and a zero eye position and (ii) shifting the depth value in a region of the depth map corresponding to the foreground image region.

11. The apparatus for generating a new viewpoint image according to claim 1 , further comprising
a filter that filters a new depth map corresponding to the new viewpoint image to smooth a depth value that is a depth value in a region of the new depth map corresponding to the missing pixel region and is a depth value less than or equal to a predetermined value.

12. The apparatus for generating a new viewpoint image according to claim 1 ,
wherein

Assignees

• Panasonic Corp
• Panasonic Ip Man Co Ltd

Inventors

• Isogai Kuniaki
• Lasang Pongsak
• Shen Sheng Mei
• Monobe Yusuke

Classifications

• G06T15/205 Primary

  Image-based rendering · CPC title

• G06T15/04 Primary

  Texture mapping · CPC title