Image processing device, image processing method, program, and integrated circuit

Patent metadata

Field	Value
Publication number	US-9401039-B2
Application number	US-201213636437-A
Country	US
Kind code	B2
Filing date	May 17, 2012
Priority date	Jul 1, 2011
Publication date	Jul 26, 2016
Grant date	Jul 26, 2016

Abstract

A depth image conversion unit moves an edge location in an input depth image towards a background. A pixel shift unit generates an image from a different viewpoint by horizontally shifting coordinates of each pixel in an image based on a converted depth image. A pixel interpolation unit interpolates a missing pixel region yielded by pixel shifting.

First claim

The invention claimed is:

1. An image processing device for generating images from multiple viewpoints, by shifting coordinates of pixels in an original image, in order to provide depth indicated by a depth image composed of a pixel region defined by a plurality of pixels, a value of each of the pixels in the depth image indicating the depth of a corresponding one of the pixels in the original image, the image processing device comprising a memory and a processor, the processor being configured to execute:
conversion processing to convert the depth image by searching within the pixel region composing the depth image for edge pixels that constitute an edge of an object and shifting, in a predetermined direction, coordinates of the pixels in the depth image that are surrounded by the edge pixels;
generation processing to generate a viewpoint image from a different viewpoint than the original image by calculating a parallax that provides the depth indicated by the depth image converted by the conversion processing and shifting the coordinates of the pixels in the original image by a number of pixels corresponding to the parallax; and
interpolation processing to interpolate an occlusion, occurring due to pixel shifting, in the generated viewpoint image, wherein:
the conversion processing includes accepting a level selection from a user when the depth image is converted, and changing the number of pixels corresponding to the parallax according to the level selection,
background pixel regions of the original image appear, due to the conversion processing, at a side to a left of the occlusion and a side to a right of the occlusion in generated viewpoint images, one of the background pixel regions between the edge of the object and the occlusion having a width according to the parallax, and
the interpolation processing includes linear interpolation using pixel values of the background pixel regions located at the side to the left of the occlusion and the side to the right of the occlusion.

2. The image processing device of claim 1 , wherein
between a foreground region and a background region separated by an edge in the depth image, the conversion processing includes shifting the coordinates of the pixels surrounded by the edge pixels towards the background region.

3. The image processing device of claim 2 , wherein:
the processor is configured to calculate a degree of complexity of an object in the original image, and
the conversion processing includes converting the depth image under a condition that the degree of complexity of the background region is at most a predetermined value.

4. The image processing device of claim 2 , wherein:
the processor is configured to calculate a degree of complexity of an object in the original image, and
the conversion processing includes converting the depth image under a condition that the degree of complexity of the foreground region is higher than the degree of complexity of the background region.

5. The image processing device of claim 2 , wherein:
the memory is configured to store a degree of complexity of the original image in a predetermined previous frame, and
the conversion processing includes converting the depth image under a condition that the degree of complexity of a foreground object in the original image in the predetermined previous frame is at least a predetermined value.

6. The image processing device of claim 1 , wherein
the conversion processing includes searching for the edge within the depth image by searching for one of the pixels whose depth value differs by at least a predetermined value from the depth value of another one of the pixels that is horizontally adjacent to the one of the pixels.

7. The image processing device of claim 6 , further comprising:
a register for storing a parameter determining the edge within the depth image, wherein
the conversion processing includes searching for the edge within the depth image by comparing a difference from the depth value of the other one of the pixels that is horizontally adjacent to the one of the pixels with the parameter stored in the register.

8. The image processing device of claim 7 , wherein:
the image processing device is configured to receive user operations, and
the parameter, stored in the register, that determines the edge within the depth image is set by user operations received by the image processing device.

9. The image processing device of claim 7 , wherein:
the processor is configured to determine a type of original image data, and
the parameter, stored in the register, that determines the edge within the depth image is based on the type of the original image data determined by the processor.

10. The image processing device of claim 6 , wherein
the conversion processing includes smoothing the pixels surrounded by the edge pixels after shifting the coordinates of the pixels surrounded by the edge pixels.

11. The image processing device of claim 1 , further comprising:
a register for storing a parameter determining a number of pixels by which to shift the coordinates of the pixels in the depth image, wherein
the conversion processing includes shifting the coordinates of the pixels in the depth image that are surrounded by the edge pixels by the number of pixels determined by the parameter and stored in the register.

12. The image processing device of claim 1 , wherein:
the processor is configured to select an image range, within the depth image, over which to perform depth image conversion, and
the conversion processing includes converting the depth image over the selected image range.

13. The image forming device of claim 1 , wherein:
the original image is a current frame from a plurality of frames in a time series;
the image processing device stores a frame processing history;
the frame processing history is data indicating, for each previous frame, whether each of the pixels in a depth image thereof has been shifted, in association with a position of the pixel in the depth image;
the searching within the pixel region composing the depth image for the edge pixels in the conversion processing includes, for each pixel of the depth image corresponding to the current frame, selecting the pixel as a current pixel, determining whether the current pixel satisfies a plurality of conditions, and updating the frame processing history;
the determining including:
(i) determining whether a depth difference between the current pixel and pixels of the depth image around the current pixel exceeds a threshold value,
(ii) determining whether complexity of foreground pixels of the depth image is greater than complexity of background pixels of the depth image, and
(iii) determining whether the frame processing history includes an indication that a pixel at the same position as the current pixel was subjected to processing in a previous frame;
wherein, when results of (i) and (ii) are negative, the frame processing history is updated to indicate that shifting was not performed at the position corresponding to the current pixel,
wherein, when results of (i) and (ii) are positive, the frame processing history is updated to indicate that shifting was performed at the position corresponding to the current pixel; and
wherein the edge pixels are composed of pixels for which the results of (i), (ii), and (iii) are positive.

14. An image processing method for generating images from multiple viewpoints, by shifting coordinates of pixels in an original image, in order to provide depth indicated in a depth image composed of a pixel region defined by a plurality of pixels, a value of each of the pixels

Assignees

• Maeda Masaki
• Ootani Naoki
• Nomura Kazuhiro
• Panasonic Ip Man Co Ltd

Inventors

• Maeda Masaki
• Ootani Naoki
• Nomura Kazuhiro

Classifications

• G06T15/205 Primary

  Image-based rendering · CPC title

• H04N13/111

  Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation · CPC title

• H04N13/128

  Adjusting depth or disparity · CPC title

• G06T15/08 Primary

  Volume rendering · CPC title

• H04N13/0022

  Electricity · mapped topic