Depth sensing using temporal coding

Patent metadata

Field	Value
Publication number	US-11941830-B2
Application number	US-202117496021-A
Country	US
Kind code	B2
Filing date	Oct 7, 2021
Priority date	Oct 2, 2018
Publication date	Mar 26, 2024
Grant date	Mar 26, 2024

Abstract

In one embodiment, a system includes at least one projector configured to project a plurality of projected patterns, where a projected lighting characteristic of each of the projected patterns varies over a time period in accordance with an associated predetermined temporal lighting-characteristic pattern, a camera configured to capture images of detected patterns during the time period, and one or more processors configured to: determine, for each detected pattern, a detected temporal lighting-characteristic pattern based on variations in a detected lighting characteristic of the detected pattern, identify a detected pattern that corresponds to one of the projected patterns by comparing at least one of the detected temporal lighting-characteristic patterns to at least one of the temporal lighting-characteristic patterns, and compute a depth associated with the detected patterns based on the one or more of the projected patterns, the detected pattern, and a relative position between the camera and the projector.

First claim

What is claimed is:

1. A method, comprising, by a computing device:
capturing, by a camera of the computing device, and during a time period, images comprising a plurality of detected patterns corresponding to reflections of a plurality of distinct projected patterns projected by a projector, wherein an intensity of each of the plurality of distinct projected patterns varies over the time period in accordance with a predetermined temporal lighting-characteristic pattern associated with the projected pattern, wherein a rate at which the images are captured by the camera is synchronized with a rate at which the intensity varies, and wherein the camera is configured to capture consecutive images over an interval of time corresponding to an interval of time between changes in the intensity of each of the plurality of distinct projected patterns;
determining, based on the images, a detected temporal lighting-characteristic pattern for each of the detected patterns;
detecting a correspondence between a first detected pattern of the detected patterns and a first projected pattern of the projected patterns by comparing the detected temporal lighting-characteristic pattern of the first detected pattern to the predetermined temporal lighting-characteristic pattern of the first projected pattern; and
computing a depth associated with the first detected pattern based on the correspondence.

2. The method of claim 1 ,
wherein computing a depth associated with the first detected pattern is based on a first detected pattern of the detected patterns, the first projected pattern of the projected patterns, and a relative position between the camera and the projector.

3. The method of claim 1 , further comprising determining a degree of difference between the detected temporal lighting-characteristic pattern of the first detected pattern and the predetermined lighting-characteristic pattern of the first projected pattern.

4. The method of claim 1 , wherein each projected pattern is projected at a different time.

5. The method of claim 1 , wherein the predetermined temporal lighting-characteristic pattern comprises three or more different lighting intensities.

6. The method of claim 1 , wherein each predetermined temporal lighting-characteristic pattern includes at least a threshold number of lighting characteristic values, and the threshold number is based on a desired level of accuracy.

7. The method of claim 1 , wherein the rate at which the intensity of each of the plurality of distinct projected patterns varies is based on an expected speed of movement of one or more objects in a scene.

8. The method of claim 1 , wherein the projected pattern has an associated density based on a distance between features in the projected pattern;
the projected pattern is projected for one or more consecutive time periods, the projected intensity of the plurality of distinct projected patterns varies over each time period in accordance with the predetermined temporal lighting-characteristic pattern associated with the projected pattern, and
the number of consecutive time periods is based on the density of the projected pattern.

9. The method of claim 1 , further comprising:
capturing, during a time period, images comprising a plurality of detected pattern projected by the projector, wherein a second detected pattern is more sparse than the first detected pattern;
determining, based on the images, the detected temporal lighting-characteristic pattern for the second detected pattern;
detecting a correspondence between a second detected pattern of the detected patterns and a second projected pattern of the projected patterns by comparing the detected temporal lighting-characteristic pattern of the second detected pattern to the predetermined temporal lighting-characteristic pattern of the second projected pattern; and
computing a depth associated with the second detected pattern based on the correspondence.

10. The method of claim 9 , wherein computing a depth associated with the second detected pattern is based on the second detected pattern of the detected patterns, the second projected pattern of the projected patterns, and a relative position between the camera and the projector.

11. The method of claim 1 , wherein the at least one projector is configured to project at least one of the plurality of distinct projected patterns repeatedly over a plurality of time periods.

12. A system, comprising:
one or more processors; and
one or more computer-readable non-transitory storage media coupled to one or more processors and comprising instructions operable when executed by one or more processors to cause the system to:
capture, by a camera of the system, and during a time period, images comprising a plurality of detected patterns corresponding to reflections of a plurality of distinct projected patterns projected by a projector, wherein an intensity of each of the plurality of distinct projected patterns varies over the time period in accordance with a predetermined temporal lighting-characteristic pattern associated with the projected pattern, wherein a rate at which the images are captured by the camera is synchronized with a rate at which the intensity varies, and wherein the camera is configured to capture consecutive images over an interval of time corresponding to an interval of time between changes in the intensity of each of the plurality of distinct projected patterns;
determine, based on the images, a detected temporal lighting-characteristic pattern for each of the detected patterns;
detect a correspondence between a first detected pattern of the detected patterns and a first projected pattern of the projected patterns by comparing the detected temporal lighting-characteristic pattern of the first detected pattern to the predetermined temporal lighting-characteristic pattern of the first projected pattern; and
compute a depth associated with the first detected pattern based on the correspondence.

13. The system of claim 12 ,
wherein the depth associated with the first detected pattern is based on a first detected pattern of the detected patterns, the first projected pattern of the projected patterns, and a relative position between the camera and the projector.

14. The system of claim 12 , wherein a degree of difference is determined between the detected temporal lighting-characteristic pattern of the first detected pattern and the predetermined lighting-characteristic pattern of the first projected pattern.

15. The system of claim 12 , wherein the rate at which the intensity of each of the plurality of distinct projected patterns varies is based on an expected speed of movement of one or more objects in a scene.

16. The system of claim 12 , wherein the first projected pattern has an associated density based on a distance between one or more features in the first projected pattern;
the first projected pattern is projected for one or more consecutive time periods, the projected intensity of the plurality distinct projected patterns varies over each time period in accordance with the predetermined temporal lighting-characteristic pattern of the first projected pattern; and
the number of consecutive time periods is based on the density of the first projected pattern.

17. One or more computer-readable non-transitory storage media embodying software that is operable when executed by one or more processors of a computing device to:
capture, by a camera of the computing device, and during a time period, images comprising a plurality of detected patterns corresponding to reflections of a plural

Assignees

• Meta Platforms Tech Llc

Inventors

• Kantor Niv
• Garcia Ricardo
• Grossinger Nadav
• Hasbun Robert
• Romano Nitay

Classifications

• G06T7/521 Primary

  from laser ranging, e.g. using interferometry; from the projection of structured light · CPC title

• G01S17/48

  Active triangulation systems, i.e. using the transmission and reflection of electromagnetic waves other than radio waves · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title