Fusing multiple depth sensing modalities

Patent metadata

Field	Value
Publication number	US-11769269-B2
Application number	US-202217878535-A
Country	US
Kind code	B2
Filing date	Aug 1, 2022
Priority date	Dec 24, 2019
Publication date	Sep 26, 2023
Grant date	Sep 26, 2023

Abstract

A method includes receiving a first depth map that includes a plurality of first pixel depths and a second depth map that includes a plurality of second pixel depths. The first depth map corresponds to a reference depth scale and the second depth map corresponds to a relative depth scale. The method includes aligning the second pixel depths with the first pixel depths. The method includes transforming the aligned region of the second pixel depths such that transformed second edge pixel depths of the aligned region are coextensive with first edge pixel depths surrounding the corresponding region of the first pixel depths. The method includes generating a third depth map. The third depth map includes a first region corresponding to the first pixel depths and a second region corresponding to the transformed and aligned region of the second pixel depths.

First claim

What is claimed is:

1. A computer-implemented method comprising:
receiving a first depth map comprising a first plurality of depth values and a second depth map comprising a second plurality of depth values;
aligning at least part of the second depth map with a corresponding part of the first depth map;
identifying a first aligned region of the first depth map having a different amount of depth information than a second aligned region of the second depth map; and
inpainting the first aligned region of the first depth map based on depth values of the second aligned region of the second depth map by determining a plurality of scaling factors comprising, for each respective depth value of a plurality of pixels in the second aligned region, a corresponding scaling factor between the respective depth value and a corresponding depth value of the first aligned region, such that an output value of an optimization function is reduced below a threshold value, wherein the optimization function is configured to compare, for each respective depth value of the plurality of pixels in the second aligned region, (i) the respective depth value scaled according to the corresponding scaling factor and (ii) the corresponding depth value of the first aligned region.

2. The computer-implemented method of claim 1 , wherein the first depth map corresponds to a first depth scale that represents the first plurality of depth values using distance units and the second depth map corresponds to a second depth scale that represents the second plurality of depth values using unitless relative distances selected from a predetermined range.

3. The computer-implemented method of claim 2 , further comprising:
determining a first sensor type used for determining the first depth map;
determining a second sensor type used for determining the second depth map;
assigning the first depth scale to the first depth map based on the first sensor type; and
assigning the second depth scale to the second depth map based on the second sensor type.

4. The computer-implemented method of claim 3 , wherein:
assigning the first depth scale to the first depth map based on determining that the first sensor type is configured to generate first sensor data that is usable to determine an absolute depth; and
assigning the second depth scale to the second depth map based on determining that the second sensor type is configured to generate second sensor data that is usable to determine a relative depth.

5. The computer-implemented method of claim 1 , wherein the first depth map comprises a multiscopic depth map determined based on multiple images captured by a multiscopic camera, and wherein the second depth map comprises a monoscopic depth map determined based on one image captured by a monoscopic camera.

6. The computer-implemented method of claim 1 , wherein the output value of the optimization function is based on a smoothness associated with the plurality of scaling factors at least in the second aligned region of the second depth map.

7. The computer-implemented method of claim 6 , wherein the optimization function is configured to measure the smoothness based on comparing (i) a first plurality of edge depth values of edge pixels of the first depth map that surround the first aligned region of the first depth map to (ii) a second plurality of edge depth values of edge pixels inside the second aligned region of the second depth map, wherein the second plurality of edge depth values has been scaled according to the corresponding scaling factors.

8. The computer-implemented method of claim 7 , wherein comparing the first plurality of edge depth values to the second plurality of edge depth values comprises determining one or more of: (i) a mean squared error therebetween, (ii) an isotropic total variation therebetween, or (iii) an anisotropic total variation therebetween.

9. The computer-implemented method of claim 1 , wherein, when the output value of the optimization function is reduced below the threshold value, a first plurality of edge depth values of edge pixels of the first depth map that surround the first aligned region of the first depth map is substantially coextensive with a second plurality of edge depth values of edge pixels inside the second aligned region of the second depth map, wherein the second plurality of edge depth values has been scaled according to the corresponding scaling factors.

10. The computer-implemented method of claim 1 , wherein the plurality of pixels in the second aligned region corresponds at least to pixels in the first aligned region for which the first depth map includes depth values.

11. The computer-implemented method of claim 1 , wherein inpainting the first aligned region of the first depth map comprises, for each respective pixel of a plurality of pixels in the first aligned region of the first depth map:
determining a corresponding inpainted depth value by multiplying a corresponding depth value in the second aligned region by the corresponding scaling factor; and
assigning the corresponding inpainted depth value to the respective pixel.

12. The computer-implemented method of claim 1 , wherein identifying the first aligned region of the first depth map comprises:
determining that the second aligned region of the second depth map has more depth information than the first aligned region of the first depth map.

13. The computer-implemented method of claim 12 , wherein determining that the second aligned region of the second depth map has more depth information than the first aligned region of the first depth map comprises:
determining one or more regions of the first depth map having less than a threshold density of depth values.

14. The computer-implemented method of claim 1 , further comprising:
generating a third depth map based on the first depth map as inpainted; and
adjusting a resolution of the third depth map such that an amount of depth information is constant throughout the third depth map.

15. A system comprising:
a processor; and
a non-transitory computer-readable medium having stored thereon instructions that, when executed by the processor, cause the processor to perform operations comprising:
receiving a first depth map comprising a first plurality of depth values and a second depth map comprising a second plurality of depth values;
aligning at least part of the second depth map with a corresponding part of the first depth map;
identifying a first aligned region of the first depth map having a different amount of depth information than a second aligned region of the second depth map; and
inpainting the first aligned region of the first depth map based on depth values of the second aligned region of the second depth map by determining a plurality of scaling factors comprising, for each respective depth value of a plurality of pixels in the second aligned region, a corresponding scaling factor between the respective depth value and a corresponding depth value of the first aligned region, such that an output value of an optimization function is reduced below a threshold value, wherein the optimization function is configured to compare, for each respective depth value of the plurality of pixels in the second aligned region, (i) the respective depth value scaled according to the corresponding scaling factor and (ii) the corresponding depth value of the first aligned region.

16. The system of claim 15 , wherein the first depth map corresponds to a first depth scale that represents the first plurality of depth values using distance units and the second depth map corresponds to a secon

Assignees

• Google Llc

Inventors

• Satat Guy
• Quinlan Michael
• Kirmani Sean
• Angelova Anelia
• Gordon Ariel

Classifications

• G06T7/593 Primary

  from stereo images · CPC title

• B25J13/089

  Determining the position of the robot with reference to its environment · CPC title

• G05D1/0231

  using optical position detecting means (position-fixing by using electromagnetic waves other than radio waves, e.g. optical position detecting means G01S5/16) · CPC title

• G06T3/20

  Linear translation of whole images or parts thereof, e.g. panning · CPC title

• H04N13/128

  Adjusting depth or disparity · CPC title