Optimized exposure control for improved depth mapping

Patent metadata

Field	Value
Publication number	US-11257237-B2
Application number	US-201916555660-A
Country	US
Kind code	B2
Filing date	Aug 29, 2019
Priority date	Aug 29, 2019
Publication date	Feb 22, 2022
Grant date	Feb 22, 2022

Abstract

Disclosed herein are optimized techniques for controlling the exposure time or illumination intensity of a depth sensor. Invalid-depth pixels are identified within a first depth map of an environment. For each invalid-depth pixel, a corresponding image pixel is identified in a depth image that was used to generate the first depth map. Multiple brightness intensities are identified from the depth image. Each brightness intensity is categorized as corresponding to either an overexposed or underexposed image pixel. An increased exposure time or illumination intensity or, alternatively, a decreased exposure time or illumination intensity is then used to capture another depth image of the environment. After a second depth map is generated based on the new depth image, portion(s) of the second depth map are selectively merged with the first depth map by replacing the invalid-depth pixels of the first depth map with corresponding valid-depth pixels of the second depth map.

First claim

What is claimed is:

1. A computer system comprising:
one or more processors; and
one or more computer-readable hardware storage devices that store computer executable instructions that are executable by the one or more processors to cause the computer system to at least:
identify invalid-depth pixels in a first depth map of an environment, including identifying, using the first depth map as a reference, x-y depth coordinates for the invalid-depth pixels, said x-y depth coordinates referring to positions in the first depth map where the invalid-depth pixels are located;
for each of at least some of the invalid-depth pixels, use the x-y depth coordinate for said each invalid-depth pixel to identify a corresponding x-y image coordinate in a first set of one or more depth image(s) that were used to generate the first depth map and to identify a corresponding image pixel located at the corresponding x-y image coordinate, including identifying a brightness intensity of said corresponding image pixel such that a plurality of brightness intensities are identified;
categorize each brightness intensity in the plurality of brightness intensities as corresponding to either an overexposed image pixel or an underexposed image pixel in the first set of one or more depth image(s), overexposed image pixels correspond to bright areas in the environment and underexposed image pixels correspond to dim areas in the environment;
as compared to a previous exposure time used when capturing the first set of one or more depth image(s), use either an increased exposure time or a decreased exposure time to capture a second set of one or more depth image(s) of the environment; and
after a second depth map is generated based on the second set of one or more depth image(s) that were captured using either the increased exposure time or the decreased exposure time, selectively merge one or more portion(s) of the second depth map with the first depth map by replacing the invalid-depth pixels of the first depth map with corresponding newly acquired valid-depth pixels of the second depth map.

2. The computer system of claim 1 , wherein, in addition to increasing or decreasing the exposure time, an illumination intensity of the depth sensor is correspondingly increased or decreased to provide additional illumination for the dim areas or, alternatively, to provide less illumination for the bright areas.

3. The computer system of claim 1 , wherein the plurality of brightness intensities are all within a range spanning between 0 and 255, brightness intensities for the overexposed image pixels are within a first threshold value of 255 and brightness intensities for the underexposed image pixels are within a second threshold value of 0.

4. The computer system of claim 3 , wherein the first threshold value is a value of 10 such that any image pixel having a corresponding brightness intensity between 245 and 255 is identified as being overexposed and is included among the overexposed image pixels, and wherein the second threshold value is also a value of 10 such that any image pixel having a corresponding brightness intensity between 0 and 10 is identified as being underexposed and is included among the underexposed image pixels.

5. The computer system of claim 1 , wherein the increased exposure time is used when the second set of one or more depth image(s) are designed to capture the dim areas of the environment and the decreased exposure time is used when the second set of one or more depth image(s) are designed to capture the bright areas of the environment.

6. The computer system of claim 1 , wherein the first set of one or more depth image(s) is captured using a stereoscopic depth camera system or, alternatively, a time of flight system.

7. The computer system of claim 6 , wherein the stereoscopic depth camera system is an active illumination stereoscopic depth camera system that illuminates using structured light.

8. The computer system of claim 6 , wherein the stereoscopic depth camera system is a passive stereoscopic depth camera system.

9. The computer system of claim 1 , wherein the invalid-depth pixels are identified as having invalid depth measurements as a result of certain image pixels, which correspond to the invalid-depth pixels, having brightness intensity values being within threshold values of either an overexposed pixel value or an underexposed pixel value.

10. The computer system of claim 1 , wherein, after the second depth map is selectively merged with the first depth map to form a newly merged depth map, execution of the computer-executable instructions further causes the computer system to:
generate a histogram plotting image pixel brightness intensities corresponding to any newly identified invalid-depth pixels identified in the newly merged depth map;
identify a number of modes in the histogram; and
based on the number of modes, generate a third depth map and selectively merge one or more portion(s) of the third depth map with the newly merged depth map in order to replace at least a majority of the newly identified invalid-depth pixels of the newly merged depth map with valid-depth pixels obtained from the third depth map.

11. The computer system of claim 1 , wherein the invalid-depth pixels of the first depth map represent that the first depth map includes one or more deficiencies with regard to mapping depths of the environment, and wherein the second depth map is generated to compensate for the one or more deficiencies of the first depth map.

12. A method for using depth data from a second depth map, which was specially designed to compensate for certain deficiencies identified within a first depth map, to thereby selectively augment said deficiencies in the first depth map, said method comprising:
identifying invalid-depth pixels in a first depth map of an environment, including identifying, using the first depth map as a reference, x-y depth coordinates for the invalid-depth pixels, said x-y depth coordinates referring to positions in the first depth map where the invalid-depth pixels are located;
for each of at least some of the invalid-depth pixels, using the x-y depth coordinate for said each invalid-depth pixel to identify a corresponding x-y image coordinate in a first set of one or more depth image(s) that were used to generate the first depth map and to identify a corresponding image pixel located at the corresponding x-y image coordinate, including identifying a brightness intensity of said corresponding image pixel such that a plurality of brightness intensities are identified;
categorizing each brightness intensity in the plurality of brightness intensities as corresponding to either an overexposed image pixel or an underexposed image pixel in the first set of one or more depth image(s), overexposed image pixels correspond to bright areas in the environment and underexposed image pixels correspond to dim areas in the environment;
as compared to a previous exposure time used when capturing the first set of one or more depth image(s), using either an increased exposure time or a decreased exposure time to capture a second set of one or more depth image(s) of the environment; and
after a second depth map is generated based on the second set of one or more depth image(s), selectively merging one or more portion(s) of the second depth map with the first depth map.

13. The method of claim 12 , wherein the increased exposure time or the decreased exposure time is selected to maximize image pixel resolutions in the second set of one or more depth image(s) for only the dim areas of the environment or the bright areas of the environment, respectively, and is selecte

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Bleyer Michael
• Pekelny Yuri
• Price Raymond Kirk

Classifications

• H04N13/128 Primary

  Adjusting depth or disparity · CPC title

• G06T7/593 Primary

  from stereo images · CPC title

• H04N23/74

  by influencing the scene brightness using illuminating means · CPC title

• H04N23/73

  by influencing the exposure time · CPC title

• H04N2013/0081

  Depth or disparity estimation from stereoscopic image signals · CPC title