Apparatus and method with depth estimation

Patent metadata

Field	Value
Publication number	US-2024179287-A1
Application number	US-202318519274-A
Country	US
Kind code	A1
Filing date	Nov 27, 2023
Priority date	Nov 25, 2022
Publication date	May 30, 2024
Grant date	—

Abstract

An apparatus and method with depth estimation are disclosed. The method includes calculating a first reliability of each of a plurality of time of flight (ToF) pixels of a ToF image; and generating, based on the first reliabilities, a depth map of a scene based on a left image and a right image and selectively based on the ToF image.

First claim

What is claimed is:

1 . A processor-implemented method of estimating depth, the method comprising:
calculating a first reliability of each of a plurality of time of flight (ToF) pixels of a ToF image; and generating, based on the first reliabilities, a depth map of a scene based on a left image and a right image and selectively based on the ToF image.

2 . The method of claim 1 , wherein the calculating of the first reliabilities comprises:
projecting each of the plurality of ToF pixels onto the left image and the right image; calculating a second reliability of a respective second ToF pixel, of the plurality of ToF pixels, corresponding to each second ToF projection point on a second corresponding scan line in a first direction; and calculating, based on the calculating of the second reliability, the first reliability of a respective first ToF pixel, of the plurality of ToF pixels, corresponding to each first ToF projection point on a first corresponding scan line in a second direction that is opposite to the first direction.

3 . The method of claim 2 , wherein the calculating of the second reliabilities and the calculating of the first reliabilities are based on an image feature difference of each second ToF projection point of the left image and the right image and a third reliability of a ToF pixel corresponding to a ToF projection point determined similar to an image feature in which a distance between each second ToF projection point is in a preset range on the second corresponding scan line.

4 . The method of claim 1 , wherein the generating of the depth map comprises:
determine a first quantity of first reliability ToF pixels, of the plurality of ToF pixels, that have respective first reliabilities that satisfy a predetermined requirement; selecting, in response to the first quantity satisfying a first threshold requirement, to generate the depth map based on the ToF image; and selecting, in response to the first quantity not satisfying the first threshold requirement, to generate the depth map without consideration of the ToF image.

5 . The method of claim 4 , wherein the generating of the depth map based on the ToF image comprises:
performing a first stereo matching of the left image and the right image, including a determination of first matched ToF pixels; predicting, using a first neural network, a fourth reliability of each of the plurality of ToF pixels based on the ToF image and a result of the first stereo matching; and generating a first depth map of the scene by performing a second stereo matching of the left image and the right image based on the ToF image and the fourth reliabilities.

6 . The method of claim 5 ,
wherein the selecting, to generate the depth map based on the ToF image, is based on the first reliabilities and the ToF image in response to a second quantity of second ToF pixels, of the plurality of ToF pixels, that have respective first reliabilities that satisfy the first threshold requirement and a second threshold requirement; or wherein the selecting, to generate the depth map without the consideration of the ToF image, is based on a third quantity of third ToF pixels, of the plurality of ToF pixels that have respective first reliabilities that satisfy the first threshold requirement and do not satisfy the second threshold requirement.

7 . The method of claim 5 , wherein the predicting of the fourth reliabilities comprises predicting the fourth reliabilities using at least one piece of information among first information, second information, and third information as an input to the first neural network,
wherein the first information is a difference between a disparity value corresponding to each of the plurality of ToF pixels and a disparity value of each of first matched ToF pixels, wherein the second information is an image feature difference of each of the plurality of ToF pixels of a corresponding projection point of the left image and the right image, and wherein the third information is a difference of depth values between the corresponding projection points and at least one ToF projection point having a determined similar feature in a corresponding projection point area.

8 . The method of claim 5 , wherein the generating of the first depth map comprises:
calculating a respective matching cost of a candidate disparity corresponding to each of the plurality of ToF pixels during the second stereo matching based on a respective value of each of the plurality of ToF pixels and the predicted fourth reliabilities of each of the plurality of ToF pixels; determining a respective disparity value corresponding to each of the plurality of ToF pixels based on the respective matching cost; and estimating the first depth map using the determined respective disparity value.

9 . The method of claim 5 , wherein the generating of the depth map comprises:
projecting the ToF image onto the left image and the right image and generating a second depth map by performing an interpolation, based on corresponding image features of the left image and the right image, on a ToF projection point area that satisfies a preset density; generating a third depth map by performing an interpolation on the first depth map based on image features of the left image and the right image; and generating a fourth depth map of the scene based on the second depth map and the third depth map.

10 . The method of claim 9 , wherein the generating of the second depth map comprises:
generating interpolated ToF projection points by respectively performing an interpolation, based on a corresponding image feature of the left image and the right image, on adjacent ToF projection points spaced apart in a preset distance on a corresponding scan line of each ToF projection point; determining a regular grid of a ToF projection point by sampling the interpolated ToF projection points; and generating the second depth map by respectively performing an interpolation, based on a respective image feature of the left image and the right image, on each ToF projection point on each determined regular grid.

11 . The method of claim 9 , wherein the performing of the interpolation of the first depth map comprises determining a depth value of a point to be interpolated based on a spatial distance and an image feature difference between a point to be interpolated and adjacent reference points.

12 . The method of claim 4 , wherein the generating of the depth map without consideration of the ToF image comprises generating a fifth depth map of the scene through a stereo matching of the left image and the right image.

13 . The method of claim 12 , further comprising:
updating a depth value of an unreliable depth value point of the generated depth map, wherein the generated depth map comprises the fifth depth map.

14 . The method of claim 13 , wherein the updating of the depth value comprises:
determining a reliable depth value point and the unreliable depth value point of the generated depth map; predicting, using a second neural network, the depth value of the unreliable depth value point based on a feature of the reliable depth value point and the unreliable depth value point; and generating an updated depth map by performing an interpolation, based on corresponding image features of the left image and the right image, on an area around the unreliable depth value point.

15 . The method of claim 14 , wherein the determining of the reliable depth value point and the unreliable depth value point comprises determining a regular grid of a depth value point based on the updated depth map,

Assignees

• Samsung Electronics Co Ltd

Inventors

• Li Weiming
• KWON Namseop
• He Bao
• Jung Kyungboo
• Jeon Myungjae
• Wang Qiang
• Sung Young Hun
• Ma Lin

Classifications

• H04N2013/0081

  Depth or disparity estimation from stereoscopic image signals · CPC title

• H04N13/239

  using two two-dimensional [2D] image sensors having a relative position equal to or related to the interocular distance (H04N13/243 takes precedence) · CPC title

• G06V20/64

  Three-dimensional [3D] objects · CPC title

• G01S17/894

  Three-dimensional [3D] imaging with simultaneous measurement of time-of-flight at a two-dimensional [2D] array of receiver pixels, e.g. time-of-flight cameras or flash lidar · CPC title

• H04N13/271 Primary

  wherein the generated image signals comprise depth maps or disparity maps · CPC title