Image processing apparatus and image processing method

Patent metadata

Field	Value
Publication number	US-9495806-B2
Application number	US-201314115647-A
Country	US
Kind code	B2
Filing date	Feb 28, 2013
Priority date	Mar 29, 2012
Publication date	Nov 15, 2016
Grant date	Nov 15, 2016

Abstract

An image processing apparatus is provided which includes: a feature calculation unit which calculates feature quantities of respective pixels included in an input image; a reliability level obtaining unit which obtains reliability level information indicating reliability levels of respective depth values indicating depths of the respective pixels; and a depth correction unit which corrects the depth values included in input depth information, using the reliability levels and the feature quantities, to generate output depth information.

First claim

The invention claimed is:

1. An image processing apparatus which corrects depth values indicating depths of pixels included in an input image, the apparatus comprising:
a feature calculation unit configured to calculate feature quantities of the pixels;
a reliability level obtaining unit configured to obtain reliability levels of the depth values; and
a depth correction unit configured to correct the depth values using the reliability levels and the feature quantities, and output corrected depth values for an output image,
wherein the correction of the depth values includes correcting a depth value of a target pixel to be processed, by preferentially using a depth value whose reliability level is high and a depth value whose feature quantity is close to a feature quantity of the depth value of the target pixel, among depth values of reference pixels included in the pixels.

2. The image processing apparatus according to claim 1 ,
wherein the depth correction unit includes:
a weight calculation unit configured to calculate weights of the reference pixels using the feature quantities and the reliability levels of the reference pixels; and
a depth synthesizing unit configured to generate the depth value of the corrected target pixel by weighted averaging the depth values of the reference pixels using the weights calculated.

3. The image processing apparatus according to claim 2 ,
wherein the weight calculation unit is configured to: calculate differences between the feature quantity of the target pixel and the feature quantities of the reference pixels;
and reduce the weight of a corresponding one of the reference pixels as the difference increases.

4. The image processing apparatus according to claim 2 ,
wherein the weight calculation unit is configured to increase the weight of a corresponding one of the reference pixels as the reliability level of the reference pixel increases.

5. The image processing apparatus according to claim 2 ,
wherein the weight calculation unit is configured to reduce the weight of a corresponding one of the reference pixels as a distance of the reference pixel from the target pixel increases.

6. The image processing apparatus according to claim 1 ,
wherein the feature quantities include edge strengths of the pixels.

7. The image processing apparatus according to claim 1 ,
wherein the feature quantities include edge directions of the pixels.

8. The image processing apparatus according to claim 1 ,
wherein the reliability level indicates one of correctness levels of the depth values and whether or not the depth values are correct.

9. The image processing apparatus according to claim 1 , further comprising
a depth estimation unit configured to estimate the depth values using a plurality of images including the input image.

10. The image processing apparatus according to claim 9 ,
wherein the depth estimation unit is configured to: calculate, for the pixels, cost values indicating matching levels between (i) a blur amount of the target pixel to be processed and (ii) a plurality of predetermined blur amounts that corresponds to a plurality of predetermined depths, the cost values being calculated for the depth values; and estimate, as the depth value of the target pixel, a depth value that corresponds to a selected cost value that indicates a highest matching level among the calculated cost values, and
the reliability level obtaining unit is configured to calculate the reliability level using the selected cost value.

11. The image processing apparatus according to claim 10 ,
wherein the reliability level obtaining unit is configured to increase the reliability level as the matching level indicated by the selected cost value increases.

12. The image processing apparatus according to claim 11 ,
wherein the reliability level obtaining unit is further configured to set the reliability level to a lowest value when the matching level indicated by the selected cost value is smaller than a threshold.

13. The image processing apparatus according to claim 12 ,
wherein the reliability level obtaining unit is further configured to change the threshold according to a luminance value of the target pixel.

14. The image processing apparatus according to claim 13 ,
wherein the reliability level obtaining unit is configured to increase the threshold as the luminance value is greater.

15. The image processing apparatus according to claim 9 , further comprising
a position alignment unit configured to move a position of an object to align the position of the object in the images,
wherein the depth estimation unit is configured to estimate the depth values using the images on which the position alignment has been performed, and
the reliability level obtaining unit is configured to calculate the reliability level using information generated in the position alignment.

16. The image processing apparatus according to claim 15 ,
wherein, in the position alignment, the position alignment unit is configured to: calculate differences between a target block included in a target image which is one of the images and a plurality of blocks included in an image different from the target image among the images; and determine a block that has a smallest difference among the calculated differences as a corresponding block that corresponds to the target block, and
the reliability level obtaining unit is configured to increase the reliability level as the difference between the target block and the corresponding block is smaller.

17. An image processing method of correcting depth values indicating depths of pixels included in an input image, the image processing method comprising:
calculating feature quantities of the pixels;
obtaining reliability levels of the depth values;
correcting the depth values using the reliability levels and the feature quantities; and
outputting corrected depth values for an output image,
wherein the correcting the depth values includes correcting a depth value of a target pixel to be processed, by preferentially using a depth value whose reliability level is high and a depth value whose feature quantity is close to a feature quantity of the depth value of the target pixel, among depth values of reference pixels included in the pixels.

18. A non-transitory computer-readable recording medium for use in a computer, the recording medium having a computer program recorded thereon for causing the computer to execute the image processing method according to claim 17 .

19. An integrated circuit which corrects depth values indicating depths of pixels included in an input image, the integrated circuit comprising:
a feature calculation unit configured to calculate feature quantities of the pixels;
a reliability level obtaining unit configured to obtain reliability levels of the depth values; and
a depth correction unit configured to correct the depth values using the reliability levels and the feature quantities, and output corrected depth values for an output image,
wherein the correction of the depth values includes correcting a depth value of a target pixel to be processed, by preferentially using a depth value whose reliability level is high and a depth value whose feature quantity is close to a feature quantity of the depth value of the target pixel, among depth values of reference pixels included in the pixels.

Assignees

• Panasonic Corp
• Panasonic Ip Man Co Ltd

Inventors

• Nguyen Khang
• Monobe Yusuke

Classifications

• G06T19/20 Primary

  Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts · CPC title

• G06T5/002

  Physics · mapped topic

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• H04N13/0011

  Electricity · mapped topic

• G06T5/001

  Physics · mapped topic