Image generation apparatus, image display apparatus, image generation method and non-transitory computer readable medium

Patent metadata

Field	Value
Publication number	US-9361725-B2
Application number	US-201313866548-A
Country	US
Kind code	B2
Filing date	Apr 19, 2013
Priority date	Apr 20, 2012
Publication date	Jun 7, 2016
Grant date	Jun 7, 2016

Abstract

Provided are an image generation apparatus, three-dimensional image display apparatus, image generation method and non-transitory computer readable medium. An image generation apparatus for generating a virtual-viewpoint image, includes a disparity-image-generating section which generates a disparity image corresponding to a two-dimensional image, by using a depth image and a parameter of a virtual camera; a disparity-image-correcting section which corrects the disparity image based on contour lines extracted from the two-dimensional image and the disparity image, or corrects the depth image to be used for generating the disparity image based on contour lines extracted from the two-dimensional image and the depth image; an pixel-shift-processing section which shifts each pixel of the two-dimensional image according to the corrected disparity image, to generate an intermediate image; and a unfixed-pixel-area-interpolating section which interpolates unfixed pixels to which the shifted pixels have been assigned in the intermediate image, to generate the virtual-viewpoint image.

First claim

The invention claimed is:

1. An image generation apparatus which uses a two-dimensional image, a depth image representing depth information of the two-dimensional image, and a parameter of a virtual camera arranged at a virtual location in a three-dimensional space, to generate a virtual-viewpoint image to be captured with the virtual camera at the virtual location, the image generation apparatus comprising:
a disparity-image-generating section which generates a disparity image representing a disparity value of each pixel of the two-dimensional image, by using the depth image and the parameter of the virtual camera;
a disparity-image-correcting section which performs one of
extracting a contour line from the two-dimensional image and a contour line from the disparity image, estimating an error area out of the disparity image generated by the disparity-image-generating section based on the extracted contour lines and correcting the disparity value of each of pixels in the error area, and
extracting a contour line from the two-dimensional image and a contour line from the depth image, estimating an error area out of the depth image based on the extracted contour lines, correcting the depth information of each of pixels in the error area and correcting the disparity image generated by the disparity-image-generating section based on the corrected depth image;

an pixel-shift-processing section which shifts each pixel of the two-dimensional image according to the disparity image which has been corrected, to generate an intermediate image; and
a unfixed-pixel-area-interpolating section which interpolates unfixed pixels to which shifted pixels of the two-dimensional image have not been assigned in the intermediate image, to generate the virtual-viewpoint image.

2. The image generation apparatus of claim 1 ,
wherein the disparity-image-correcting section, for estimating the error area,
defines a pixel of the contour line of the two-dimensional image as a reference pixel, the pixel being located on a predetermined one of lines which are one of rows and columns,
determines two pixels of the contour line of the disparity image such that the two pixels are separated in a positive direction and a negative direction along the other of the rows and columns at a maximum distance within an area including lines which are the one of rows and columns in a vicinity of the predetermined one of lines, and
estimates an area located between the two pixels and extending along the other of rows and columns as the error area.

3. The image generation apparatus of claim 1 ,
wherein the disparity-image-correcting section, for estimating the error area,
defines a pixel of the contour line of the two-dimensional image as a reference pixel, the pixel being located on a predetermined one of lines which are one of rows and columns,
determines two pixels of the contour line of the depth image such that the two pixels are separated in a positive direction and a negative direction along the other of the rows and columns at a maximum distance within an area including lines which are the one of rows and columns in a vicinity of the predetermined one of lines, and
estimates an area located between the two pixels and extending along the other of rows and columns as the error area.

4. An image generation apparatus which uses a plurality of two-dimensional images to generate a virtual-viewpoint image to be captured with a virtual camera arranged at a virtual location in a three-dimensional space, the image generation apparatus comprising:
a disparity-image-generating section which generates a disparity image representing a disparity value of each pixel of one of the plurality of two-dimensional image, by using the plurality of two-dimensional images;
a disparity-image-correcting section which extracts a contour line from the one of the two-dimensional images and a contour line from the disparity image, estimates an error area out of the disparity image generated by the disparity-image-generating section based on the extracted contour lines and corrects the disparity value of each of pixels in the error area;
a pixel-shift-processing section which shifts each pixel in the one of the plurality of two-dimensional images according to the disparity image which has been corrected, to generate an intermediate image; and
a unfixed-pixel-area-interpolating section which interpolates unfixed pixels to which shifted pixels of the one of the two-dimensional image have not been assigned in the intermediate image, to generate the virtual-viewpoint image.

5. The image generation apparatus of claim 4 ,
wherein the disparity-image-correcting section, for estimating the error area,
defines a pixel of the contour line of the one of the two-dimensional images as a reference pixel, the pixel being located on a predetermined one of lines which are one of rows and columns,
determines two pixels of the contour line of the disparity image such that the two pixels are separated in a positive direction and a negative direction along the other of the rows and columns at a maximum distance within an area including lines which are the one of rows and columns in a vicinity of the predetermined one of lines,
estimates an area located between the two pixels and extending along the other of rows and columns as the error area.

6. An image generation apparatus which uses a two-dimensional image, a depth image representing depth information of the two-dimensional image, and a parameter of a virtual camera arranged at a virtual location in a three-dimensional space, to generate a virtual-viewpoint image to be captured with the virtual camera at the virtual location, the image generation apparatus comprising:
a disparity-image-generating section which generates a disparity image representing a disparity value of each pixel of the two-dimensional image, by using the depth image and the parameter of the virtual camera;
a pixel-shift-processing section which shifts each pixel of the two-dimensional image according to the disparity image, to generate an intermediate image;
a noise-in-unfixed-pixel-area-eliminating section which eliminates noise in an area of unfixed pixels to which shifted pixels of the two-dimensional image have not been assigned in the intermediate image; and
a unfixed-pixel-area-interpolating section which interpolates the unfixed pixels in the intermediate image wherein the noise has been eliminated, to generate the virtual-viewpoint image.

7. The image generation apparatus of claim 6 , wherein
the noise-in-unfixed-pixel-area-eliminating section comprises
a noise-judging section which compares a length of a pixel area located between two areas of unfixed pixels and a predetermined threshold for judging noise, to judge whether the pixel area corresponds to noise or not, and
a noise-eliminating section which performs a noise elimination in the pixel area when the pixel area is judged as noise.

8. The image generation apparatus of claim 6 , wherein
the noise-in-unfixed-pixel-area-eliminating section comprises
a noise-judging section which compares a ratio of a total length of two areas of unfixed pixels to a length of a pixel area located between the two areas of unfixed pixels, and a predetermined threshold for judging noise, to judge whether the pixel area corresponds to noise or not, and
a noise-eliminating section which performs a noise elimination in the pixel area when the pixel area is judged as noise.

9. An image generation apparatus which uses a plurality of two-dimensional images to generate a virtual-viewpoint image to be captured with a virtual camera arranged at a virtual location in a three-dimensional space, the image generation apparatus comprising:

Assignees

• Nlt Technologies Ltd

Inventors

• Wu Di
• Sakamoto Michiaki

Classifications

• H04N13/0271

  Electricity · mapped topic

• G06T15/00 Primary

  Three-dimensional [3D] image rendering · CPC title

• H04N13/0011

  Electricity · mapped topic

• H04N13/271

  wherein the generated image signals comprise depth maps or disparity maps · CPC title

• H04N13/111 Primary

  Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation · CPC title