Image processing device and image processing method

Patent metadata

Field	Value
Publication number	US-10846826-B2
Application number	US-201716089822-A
Country	US
Kind code	B2
Filing date	Jan 16, 2017
Priority date	Apr 6, 2016
Publication date	Nov 24, 2020
Grant date	Nov 24, 2020

Abstract

An input image information generation processing section 21 generates depth distribution feature information in accordance with the pixel value and depth value of a pixel in a peripheral region with respect to a pixel of interest of an input image. A previous image information generation processing section 24 regards the pixel position of a pixel of interest in a previous image, which is an image earlier than the input guide image, as a corresponding pixel, and generates depth distribution feature information in accordance with the pixel value of the pixel of interest and the pixel value and depth value of a pixel in a previous image peripheral region with respect to the corresponding pixel. A merge control section 25 sets a merge ratio in accordance with the amount of pixel information difference between the pixel of interest and the corresponding pixel. A depth value setting section 26 merges the depth distribution feature information generated by the input image information generation processing section 21 with the depth distribution feature information generated by the previous image information generation processing section 24 at the merge ratio set by the merge control section 25, calculates a representative depth value from the merged depth distribution feature information, and regards the calculated representative depth value as the depth value of the pixel of interest. Spatiotemporal stability of a depth image can be improved.

First claim

The invention claimed is:

1. An image processing device, comprising:
an input image information generation processing section configured to generate input image depth distribution feature information from an input image based on first pixel values and first depth values of first pixels in a first peripheral region of the input image with respect to a pixel of interest of the input image;
a previous image information generation processing section configured to generate previous image depth distribution feature information from a previous image generated before the input image, wherein the previous image depth distribution feature information is generated based on a pixel value of the pixel of interest and second pixel values and second depth values of second pixels in a second peripheral region of the previous image with respect to a previous pixel that corresponds to the pixel of interest; and
a depth value processing section configured to:
merge the input image depth distribution feature information with the previous image depth distribution feature information to generate merged depth distribution feature information;
calculate a representative depth value from the merged depth distribution feature information;
determine that a reliability of the representative depth value is not lower than a preset determination threshold value; and
regard the calculated representative depth value as a depth value of the pixel of interest based on the determination that the reliability of the representative depth value is not lower than the preset determination threshold value.

2. The image processing device according to claim 1 , wherein
the input image information generation processing section is further configured to generate, for each pixel of the first pixels in the first peripheral region, the input image depth distribution feature information based on association of the first depth values with a first weight,
the first weight is based on a similarity of pixel information of the each pixel of the first pixels in the first peripheral region to that of the pixel of interest,
the previous image information generation processing section is further configured to generate, for each pixel of the second pixels in the second peripheral region with respect to the previous pixel, the previous image depth distribution feature information based on association of the second depth values with a second weight, and
the second weight is based on a similarity of the pixel information of the each pixel of the second pixels in the second peripheral region to that of the pixel of interest.

3. The image processing device according to claim 2 , further comprising:
a merge control section configured to set a merge ratio based on an amount of pixel information difference between the pixel of interest and a corresponding pixel,
wherein the depth value processing section is further configured to merge the input image depth distribution feature information with the previous image depth distribution feature information at the set merge ratio.

4. The image processing device according to claim 3 , wherein the merge control section is further configured to increase the merge ratio for the input image depth distribution feature information with an increase in the amount of pixel information difference between the pixel of interest and the corresponding pixel.

5. The image processing device according to claim 3 , wherein the pixel information includes information associated with luminance or color.

6. The image processing device according to claim 2 , wherein
the input image information generation processing section is further configured to increase the first weight with an increase in the similarity of the pixel information of the each pixel of the first pixels in the first peripheral region to that of the pixel of interest, and
the previous image information generation processing section is further configured increase the second weight with an increase in the similarity of the pixel information of the each pixel of the second pixels in the second peripheral region to that of the pixel of interest.

7. The image processing device according to claim 2 , wherein the depth value processing section is further configured to regard, as the reliability, a ratio of a total weight in a specific depth range based on the representative depth value to a total weight of the merged depth distribution feature information.

8. The image processing device according to claim 2 , wherein
the depth value processing section is further configured to regard the depth value of the pixel of interest as an invalid value based on the reliability of the representative depth value being lower than the preset determination threshold value, and
the previous image information generation processing section is further configured to generate the previous image depth distribution feature information by exclusion of pixels indicative of the invalid value from the second pixels.

9. The image processing device according to claim 2 , wherein the depth value processing section is further configured to change the preset determination threshold value.

10. The image processing device according to claim 2 , wherein the depth value processing section is further configured to:
regard the merged depth distribution feature information as depth value order information; and
set a depth value having a median accumulated weight value as the representative depth value.

11. The image processing device according to claim 2 , wherein
the input image information generation processing section is further configured to:
accumulate the first weight of each depth value of the first depth values;
obtain a first depth histogram based on the accumulation of the first weight;
generate the first depth histogram as the depth distribution feature information,

the previous image information generation processing section is further configured to:
accumulate the second weight of each depth value of the second depth values;
obtain a second depth histogram based on the accumulation of the second weight;
generate the second depth histogram as the previous image depth distribution feature information, and

the depth value processing section is further configured to:
merge the first depth histogram and the second depth histogram based on the set merge ratio and weights of the first depth histogram and the second depth histogram; and
regard the merged first depth histogram and the second depth histogram as the merged depth distribution feature information.

12. The image processing device according to claim 2 , wherein
the input image information generation processing section is further configured to generate, as the depth distribution feature information, a first depth table,
the first depth table indicates the first weight of each depth value of the first depth values,
the previous image information generation processing section is further configured to generate, as the depth distribution feature information, a second depth table,
the second depth table indicates the second weight of each depth value of the second depth values, and
the depth value processing section is further configured to:
merge the first depth table and the second depth table based on the set merge ratio and weight of the first depth table and the second depth table; and
regard the merged first depth table and the second depth table as the merged depth distribution feature information.

13. The image processing device according to claim 2 , wherein
the depth value processing section is further configured to regard, as the reliability of the pixel of in

Assignees

• Sony Corp

Inventors

• Myokan Yoshihiro

Classifications

• G06F18/22

  Matching criteria, e.g. proximity measures · CPC title

• G06T1/00

  General purpose image data processing · CPC title

• H04N13/261

  with monoscopic-to-stereoscopic image conversion · CPC title

• H04N13/128 Primary

  Adjusting depth or disparity · CPC title

• G06T7/50 Primary

  Depth or shape recovery · CPC title