Different-view image generating apparatus and different-view image generating method

Patent metadata

Field	Value
Publication number	US-9596445-B2
Application number	US-201314374379-A
Country	US
Kind code	B2
Filing date	Oct 16, 2013
Priority date	Nov 30, 2012
Publication date	Mar 14, 2017
Grant date	Mar 14, 2017

Abstract

An apparatus includes: a generating unit which generates, from each of images respectively obtained at viewpoint positions, one of different-viewpoint images which corresponds to an image at a virtual viewpoint position different from the viewpoint positions, the different-viewpoint image including a hole area in which a pixel value is missing; a calculating unit which calculates, for each of processing units respectively in predetermined areas in the different-viewpoint images, a hole density indicating, with respect to the predetermined area, a ratio of one of the hole areas in the processing units in the different-viewpoint images; a calculating unit which calculates, for each processing unit, a combination ratio of the different-viewpoint image, based on the hole density of the processing unit co-located with an other one of the processing units in an other one of the different-viewpoint images; and a combining unit which combines the different-viewpoint images, based on the combination ratios.

First claim

The invention claimed is:

1. A different-viewpoint image generating apparatus comprising:
a processor; and
a non-transitory computer-readable medium having stored thereon executable instructions that, when executed by the processor, cause the different-viewpoint image generating apparatus to function as:
a different-viewpoint image generating unit configured to generate, from each of two or more images respectively obtained at two or more viewpoint positions, one of two or more different-viewpoint images which correspond to an image obtainable at a virtual viewpoint position different from the two or more viewpoint positions, based on distance information indicating a depth of a pixel in one of the two or more images, the one of the two or more different-viewpoint images including a hole area in which a pixel value is missing;
a hole density calculating unit configured to calculate, for each of processing units respectively in predetermined areas in the two or more different-viewpoint images, a hole density indicating, with respect to the predetermined area, a ratio of one of the hole areas in the processing units in the two or more different-viewpoint images, the processing unit being made up of one or more pixels;
a combination ratio calculating unit configured to calculate, for each of the processing units, a combination ratio of one of the two or more different-viewpoint images, based on the hole density of the processing unit which is co-located with an other one of the processing units in an other one of the two or more different-viewpoint images; and
a different-viewpoint image combining unit configured to combine the two or more different-viewpoint images, based on the combination ratios of the processing units,
wherein the hole density calculating unit is configured to calculate, as a plurality of the hole densities of the co-located processing units in the two or more different-viewpoint images, a plurality of the ratios of the hole areas in windows which are the predetermined areas having the co-located processing units as centers of the predetermined areas, and
wherein the hole density calculating unit is configured to calculate a plurality of the hole densities by assigning different weights to a hole area located at a central part of the window and a hole area located at a peripheral part of the window.

2. The different-viewpoint image generating apparatus according to claim 1 ,
wherein the executable instructions, when executed by the processor, cause the different-viewpoint image generating apparatus to further function as a hole area interpolating unit configured to interpolate each of the hole areas in the two or more different-viewpoint images, using pixel values within a corresponding one of the two or more different-viewpoint images,
wherein the different-viewpoint image combining unit is configured to combine the two or more different-viewpoint images including the interpolated hole areas, based on the combination ratios.

3. The different-viewpoint image generating apparatus according to claim 1 ,
wherein the executable instructions, when executed by the processor, cause the different-viewpoint image generating apparatus to further function as a hole area interpolating unit configured to interpolate the hole areas within images combined by the different-viewpoint image combining unit, using pixel values within the images.

4. The different-viewpoint image generating apparatus according to claim 1 ,
wherein the combination ratio calculating unit is configured to calculate, for the co-located processing units in the two or more different-viewpoint images, a plurality of the combination ratios which become larger as a plurality of the hole densities of the co-located processing units become smaller.

5. The different-viewpoint image generating apparatus according to claim 1 ,
wherein, when hole densities of co-located processing units are equal to each other between the two or more different-viewpoint images, the combination ratio calculating unit is configured to calculate, for the co-located processing units having the equal hole densities, combination ratios which become larger as the positions of the two or more different-viewpoint images become closer to the position of the virtual viewpoint position.

6. The different-viewpoint image generating apparatus according to claim 1 ,
wherein the processing unit is a pixel,
the hole density calculating unit is configured to calculate a hole density for each of pixels co-located in the two or more different-viewpoint images, and
the combination ratio calculating unit is configured to calculate, for each of the co-located pixels, the combination ratio of one of the two or more different-viewpoint images.

7. A different-viewpoint image generating method comprising:
generating, from each of two or more images respectively obtained at two or more viewpoint positions, one of two or more different-viewpoint images which correspond to an image obtainable at a virtual viewpoint position different from the two or more viewpoint positions, based on distance information indicating a depth of a pixel in one of the two or more images, the one of the two or more different-viewpoint images including a hole area in which a pixel value is missing;
calculating, for each of processing units respectively in predetermined areas in the two or more different-viewpoint images, a hole density indicating, with respect to the predetermined area, a ratio of one of the hole areas in the processing units in the two or more different-viewpoint images, the processing unit being made up of one or more pixels;
calculating, for each of the processing units, a combination ratio of one of the two or more different-viewpoint images, based on the hole density of the processing unit which is co-located with an other one of the processing units in an other one of the two or more different-viewpoint images; and
combining the two or more different-viewpoint images, based on the combination ratios of the processing units,
wherein the calculating of the hole density comprises calculating, as a plurality of the hole densities of the co-located processing units in the two or more different-viewpoint images, a plurality of the ratios of the hole areas in windows which are the predetermined areas having the co-located processing units as centers of the predetermined areas, and
wherein, in the calculating of the hole density, a plurality of the hole densities are calculated by assigning different weights to a hole area located at a central part of the window and a hole area located at a peripheral part of the window.

8. A non-transitory computer-readable recording medium for use in a computer, the recording medium having a computer program recorded thereon for causing the computer to execute the different-viewpoint image generating method according to claim 7 .

9. An integrated circuit comprising:
a different-viewpoint image generating circuit configured to generate, from each of two or more images respectively obtained at two or more viewpoint positions, one of two or more different-viewpoint images which correspond to an image obtainable at a virtual viewpoint position different from the two or more viewpoint positions, based on distance information indicating a depth of a pixel in one of the two or more images, the one of the two or more different-viewpoint images including a hole area in which a pixel value is missing;
a hole density calculating circuit configured to calculate, for each of processing units respectively in predetermined areas in the two or more different-viewpoint images, a hole density indicating, with respect to the predetermined area, a ratio of one of the hole areas in the processi

Assignees

• Panasonic Corp
• Panasonic Ip Man Co Ltd

Inventors

• Isogai Kuniaki
• Monobe Yusuke

Classifications

• H04N13/111 Primary

  Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation · CPC title

• H04N13/128

  Adjusting depth or disparity · CPC title

• G06T15/205

  Image-based rendering · CPC title

• H04N13/0022

  Electricity · mapped topic

• H04N13/0011 Primary

  Electricity · mapped topic