Active stereo matching for depth applications

Patent metadata

Field	Value
Publication number	US-12075020-B2
Application number	US-202217981781-A
Country	US
Kind code	B2
Filing date	Nov 7, 2022
Priority date	Mar 22, 2018
Publication date	Aug 27, 2024
Grant date	Aug 27, 2024

Abstract

A head-mounted device (HMD) is configured to perform depth detection with a stereo camera pair comprising a first camera and a second camera, both of which are configured to detect/capture visible light and IR light. The fields of view for both of the cameras overlap to form an overlapping field of view. The HMD also includes an IR dot-pattern illuminator that is mounted on the HMD with the cameras and that is configured to emit an IR dot-pattern illumination. The IR dot-pattern illuminator emits a dot-pattern illumination that spans at least a part of the overlapping field of view. The IR dot-pattern illumination adds texture to objects in the environment and enables the HMD to determine depth for those objects, even if they have textureless/smooth surfaces.

First claim

What is claimed is:

1. A system that adjusts a depth map in an attempt to eliminate a potential jitter effect caused when a difference in depth values between neighboring pixels exceeds a difference threshold, the system comprising:
at least one processor; and
at least one hardware storage device that stores instructions that are executable by the at least one processor to cause the system to:
access a depth map that includes a plurality of pixels comprising depth values, wherein a first set of the pixels in the depth map correspond to a virtual object that is generated by the system such that the first set of pixels correspond to virtualized image content, and wherein a second set of the pixels in the depth map correspond to a real-world object that is included in an environment in which the system is operating such that the second set of pixels correspond to real-world content;
identify, from within the depth map, a pair of neighboring pixels comprising a first pixel having a first depth value and a second pixel having a second depth value, wherein the first pixel is included in the first set of pixels corresponding to the virtualized content, and wherein the second pixel is included in the second set of pixels corresponding to the real-world content;
determine that a difference between the first depth value and the second depth value exceeds a predefined difference threshold, wherein said difference indicates that a depth of the virtual object is different than a depth of the real-world object relative to the system; and
based on said determination that the difference between the first depth value and the second depth value exceeds the predefined difference threshold, adjust the depth map to eliminate a potential for a jitter effect for the virtual object, where the jitter effect involves the system repeatedly adjusting the depth of the virtual object in an attempt to account for the difference between the depth of the virtual object and the depth of the real-world object.

2. The system of claim 1 , wherein adjusting the depth map includes replacing the second depth value with a new depth value.

3. The system of claim 2 , wherein the new depth value is selected such that a difference between the new depth value and the first depth value does not exceed the predefined difference threshold.

4. The system of claim 1 , wherein adjusting the depth map includes imposing a filter on the depth map.

5. The system of claim 1 , wherein the system includes a pattern illuminator that illuminates an observable pattern.

6. The system of claim 5 , wherein an image of the observable pattern is stored as a reference.

7. The system of claim 1 , wherein at least one of the first pixel or the second pixel is partially representative of an object that includes a textureless surface.

8. The system of claim 1 , wherein the depth map is generated using stereo depth imaging.

9. A method for adjusting a depth map in an attempt to eliminate a potential jitter effect caused when a difference in depth values between neighboring pixels exceeds a difference threshold, the method comprising:
accessing a depth map that includes a plurality of pixels comprising depth values, wherein a first set of the pixels in the depth map correspond to a virtual object such that the first set of pixels correspond to virtualized image content, and wherein a second set of the pixels in the depth map correspond to a real-world object such that the second set of pixels correspond to real-world content;
identifying, from within the depth map, a pair of neighboring pixels comprising a first pixel having a first depth value and a second pixel having a second depth value, wherein the first pixel is included in the first set of pixels corresponding to the virtualized content, and wherein the second pixel is included in the second set of pixels corresponding to the real-world content;
determining that a difference between the first depth value and the second depth value exceeds a predefined difference threshold, wherein said difference indicates that a depth of the virtual object is different than a depth of the real-world object; and
based on said determination that the difference between the first depth value and the second depth value exceeds the predefined difference threshold, adjusting the depth map to eliminate a potential for a jitter effect for the virtual object, where the jitter effect involves a repeated adjustment of the depth of the virtual object in an attempt to account for the difference between the depth of the virtual object and the depth of the real-world object.

10. The method of claim 9 , wherein a geometric surface is constructed for an environment using the depth map.

11. The method of claim 9 , wherein the method is performed by a mixed-reality (MR) system.

12. The method of claim 9 , wherein adjusting the depth map includes replacing the first depth value with a new depth value.

13. The method of claim 12 , wherein the new depth value is selected such that a difference between the new depth value and the second depth value does not exceed the predefined difference threshold.

14. The method of claim 9 , wherein a filter is used to adjust the depth map.

15. The method of claim 9 , wherein the depth map is generated based on a plurality of images that are generated by head tracking cameras.

16. The method of claim 9 , wherein the depth map is generated based on a plurality of images that are generated by hand tracking cameras.

17. A method for adjusting a depth map in an attempt to eliminate a potential jitter effect caused when a difference in depth values between neighboring pixels exceeds a difference threshold, the method comprising:
accessing a depth map that includes a plurality of pixels comprising depth values, wherein a first set of the pixels in the depth map correspond to a virtual object such that the first set of pixels correspond to virtualized image content, and wherein a second set of the pixels in the depth map correspond to a real-world object such that the second set of pixels correspond to real-world content;
identifying, from within the depth map, a set of pixels comprising a first pixel having a first depth value and a second pixel having a second depth value, wherein the first pixel is included in the first set of pixels corresponding to the virtualized content, and wherein the second pixel is included in the second set of pixels corresponding to the real-world content;
determining that a difference between the first depth value and the second depth value exceeds a predefined difference threshold, wherein said difference indicates that a depth of the virtual object is different than a depth of the real-world object; and
based on said determination that the difference between the first depth value and the second depth value exceeds the predefined difference threshold, adjusting the depth map to eliminate a potential for a jitter effect for the virtual object, where the jitter effect involves a repeated adjustment of the depth of the virtual object in an attempt to account for the difference between the depth of the virtual object and the depth of the real-world object.

18. The method of claim 17 , wherein adjusting the depth map includes replacing the second depth value with a new depth value, and wherein the new depth value is selected such that a difference between the new depth value and the first depth value does not exceed the predefined difference threshold.

19. The method of claim 17 , wherein the first pixel is a neighbor to the second p

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Bleyer Michael
• Pekelny Yuri
• Price Raymond Kirk

Classifications

• H04N23/11

  for generating image signals from visible and infrared light wavelengths · CPC title

• G02B2027/0138

  comprising image capture systems, e.g. camera · CPC title

• G09G3/025

  with scanning or deflecting the beams in two directions or dimensions · CPC title

• G06F3/0346

  with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors · CPC title

• G02B2027/014

  comprising information/image processing systems · CPC title