Image processing apparatus and method for determining a depth of a pixel of a reference image

Patent metadata

Field	Value
Publication number	US-10692235-B2
Application number	US-201715655841-A
Country	US
Kind code	B2
Filing date	Jul 20, 2017
Priority date	Apr 28, 2015
Publication date	Jun 23, 2020
Grant date	Jun 23, 2020

Abstract

The disclosure relates to an image processing apparatus for determining a depth of a pixel of a reference image of a plurality of images representing a visual scene relative to a plurality of locations, wherein the plurality of locations define a two-dimensional grid with rows and columns and wherein the location of the reference image is associated with a reference row and a reference column of the grid. The image processing apparatus comprises a depth determiner configured to determine a first depth estimate on the basis of the reference image and a first subset of the plurality of images for determining the depth of the pixel of the reference image, wherein the images of the first subset are associated with locations being associated with a row of the grid different than the reference row and with a column of the grid different than the reference column.

First claim

What is claimed is:

1. An image processing apparatus, comprising a processor and a non-transitory computer-readable medium having processor-executable instructions stored thereon, wherein the processor-executable instructions, when executed, facilitate:
determining a depth of a pixel of a reference image of a plurality of images representing a visual scene relative to a plurality of locations, wherein each image of the plurality of images is associated with a respective location of the plurality of locations, wherein the plurality of locations define a two-dimensional grid with rows and columns, wherein the location of the reference image is associated with a reference row and a reference column of the grid, and wherein determining the depth of the pixel of the reference image comprises: determining a first depth estimate based on the reference image and a first subset of the plurality of images, wherein images of the first subset of the plurality of images include at least one image associated with a location in a row of the grid different than the reference row and at least one image associated with a location in a column of the grid different than the reference column;
wherein determining the first depth estimate comprises:
determining a modified horizontal plane by removing images of the first subset of the plurality of images that are not to be processed from a horizontal plane, wherein the pixel of the reference image and corresponding pixels in other images of the first subset of the plurality of images define a line along the horizontal plane;
refocusing the modified horizontal plane to determine a refocused modified horizontal plane;
determining an angle defined by an epipolar line in the refocused modified horizontal plane; and
determining the first depth estimate based on the angle defined by the epipolar line;

wherein the processor-executable instructions, when executed, further facilitate: determining a confidence value associated with the first depth estimate, wherein determining the confidence value for the first depth estimate excludes a depth estimate based on a filter K defined by the equation:

K
⁡

(
d
)

=

e

-

d
2

2
⁢
π

⁢
b

,

where d denotes the disparity of the pixel and b denotes the baseline defined by the reference image and the first subset of the plurality of images.

2. The image processing apparatus of claim 1 , wherein determining the depth of the pixel of the reference image further comprises:
determining a second depth estimate based on the reference image and a second subset of the plurality of images, wherein images of the second subset of the plurality of images are associated with locations in the reference row; and
combining the first depth estimate and the second depth estimate.

3. The image processing apparatus of claim 1 , wherein determining the depth of the pixel of the reference image further comprises:
determining a second depth estimate based on the reference image and a second subset of the plurality of images, wherein images of the second subset of the plurality of images are associated with locations in the reference row;
determining a third depth estimate based on the reference image and a third subset of the plurality of images, wherein images of the third subset of the plurality of images are associated with locations in the reference column; and
combining at least two of the first depth estimate, the second depth estimate or the third depth estimate.

4. The image processing apparatus of claim 3 , wherein determining the depth of the pixel of the reference image further comprises:
computing a median value of a set of values comprising at least one of the first depth estimate, the second depth estimate or the third depth estimate.

5. The image processing apparatus of claim 1 , wherein determining the depth of the pixel of the reference image further comprises:
determining a second depth estimate based on the reference image and a second subset of the plurality of images, wherein images of the second subset of the plurality of images are associated with locations being associated with the reference row; and
determining a third depth estimate based on the reference image and a third subset of the plurality of images, wherein images of the third subset of the plurality of images are associated with locations being associated with the reference column.

6. The image processing apparatus of claim 1 , wherein determining the depth of the pixel of the reference image further comprises:
determining a second depth estimate based on the reference image and a second subset of the plurality of images, wherein images of the second subset of the plurality of images are associated with locations being associated with the reference row; and
determining a third depth estimate based on the reference image and a third subset of the plurality of images, wherein images of the third subset of the plurality of images are associated with locations being associated with the reference column;
wherein the processor-executable instructions, when executed, further facilitate: determining respective confidence values associated with the second depth estimate and the third depth estimate;
wherein determining the depth of the pixel of the reference image further comprises choosing as the depth of the pixel of the reference image the depth estimate having the largest confidence value from the group consisting of the first depth estimate, the second depth estimate and the third depth estimate.

7. The image processing apparatus of claim 5 , wherein determining the confidence value for the first depth estimate, the second depth estimate or the third depth estimate is based on a structure tensor defined by the first subset, the second subset or the third subset of the plurality of images, respectively.

8. The image processing apparatus of claim 1 , wherein determining the depth of the pixel of the reference image is based on the first depth estimate and further comprises interpolating the intensity of the pixel between the intensity of the pixel in the reference image and the intensities of the corresponding pixels in the first subset of the plurality of images.

9. The image processing apparatus of claim 1 , wherein the image processing apparatus further comprises an image recorder configured to record the plurality of images representing the visual scene at the plurality of locations.

10. The image processing apparatus of claim 9 , wherein the image recorder comprises a movable camera, an array of cameras or a plenoptic camera.

11. The image processing apparatus of claim 1 , wherein the processor-executable instructions, when executed, further facilitate:
determining a depth map for the reference image by determining respective depths for a plurality of pixels of the reference image.

12. The image processi

Assignees

• Huawei Tech Co Ltd

Inventors

• Papadhimitri Thoma
• Urfalioglu Onay
• Navarro Fructuoso Hector
• Konieczny Jacek

Classifications

• H04N23/90

  Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums · CPC title

• G06T7/596

  from three or more stereo images · CPC title

• G06T7/557 Primary

  from light fields, e.g. from plenoptic cameras · CPC title

• G06T2207/10052

  Images from lightfield camera · CPC title

• G06T2200/21

  involving computational photography · CPC title