Depth Sensing Using Grid Light Patterns

Patent metadata

Field	Value
Publication number	US-2020105005-A1
Application number	US-201816150210-A
Country	US
Kind code	A1
Filing date	Oct 2, 2018
Priority date	Oct 2, 2018
Publication date	Apr 2, 2020
Grant date	—

Abstract

In one embodiment, a system includes a first projector configured to project a first projected pattern having one or more first projected lighting characteristics, a second projector configured to project a second projected pattern having one or more second projected lighting characteristics, a camera configured to capture an image comprising first and second detected patterns corresponding to reflections of the first and second projected patterns, respectively, and one or more processors configured to: identify a detected point in the image that corresponds to a projected point in at least one of the first and second projected patterns by comparing detected lighting characteristics of the first and second detected patterns with the first and second projected lighting characteristics, and compute a depth associated with the detected point based on the projected point, the detected point, and a relative position between the camera and at least one of the projectors.

First claim

What is claimed is:

1 . A system comprising:
a first projector configured to project a first projected pattern having one or more first projected lighting characteristics; a second projector configured to project a second projected pattern having one or more second projected lighting characteristics, wherein the first projected pattern intersects the second projected pattern; a camera configured to capture an image comprising first and second detected patterns corresponding to reflections of the first and second projected patterns, respectively, wherein the camera is located at a first baseline distance from the first projector and at a second baseline distance from the second projector; and one or more processors configured to:
identify a detected point in the image that corresponds to a projected point in at least one of the first and second projected patterns by comparing detected lighting characteristics of the first and second detected patterns with the first and second projected lighting characteristics; and
compute a depth associated with the detected point based on the projected point, the detected point, and a relative position between the camera and at least one of the first and second projectors.

2 . The system of claim 1 , wherein the first projected pattern comprises a plurality of first projected lines, and the second projected pattern comprises a plurality of second projected lines, and
wherein the detected lighting characteristics comprise a plurality of first reflected lines and a plurality of second reflected lines that intersect the first reflected lines, and the first and second reflected lines are based on reflections of the first and second projected lines from a surface of an object.

3 . The system of claim 2 , wherein to identify a detected point in the image that corresponds to a projected point in one or more of the first and second projected patterns, the processors are further configured to:
identify a reflected junction at which one of the first reflected lines intersects one of the second reflected lines, wherein the reflected junction is associated with one or more reflected junction characteristics; identify a projected junction that corresponds to the reflected junction, wherein the projected junction is at an intersection of one of the first projected lines and one of the second projected lines, wherein the projected junction is associated with one or more projected junction characteristics determined based on (1) the one or more first projected lighting characteristics associated with the first projected pattern and (2) the one or more second projected lighting characteristics associated with the second projected pattern, and wherein one or more of the projected junction characteristics match one or more of the reflected junction characteristics.

4 . The system of claim 3 , wherein to compute a depth for the detected point, the processors are further configured to determine, using triangulation, a depth associated with the reflected junction based on: the camera, at least one of the projectors, the reflected junction, and the projected junction that corresponds to the reflected junction.

5 . The system of claim 3 , wherein at least one of the projected junction characteristics of the projected junction matches at least one of the reflected junction characteristics when a value of the reflected junction characteristic:
is equal to a value of the projected junction characteristic, or differs from the value of the projected junction characteristic by less than a threshold characteristic value.

6 . The system of claim 5 , wherein the threshold characteristic value comprises a predetermined value or a percentage of the value of the projected junction characteristic.

7 . The system of claim 3 , wherein the projected junction characteristics and the reflected junction characteristics each comprises a light intensity.

8 . The system of claim 7 , wherein at least one of the projected junction characteristics of the projected junction matches at least one of the reflected junction characteristics when:
(1) a light intensity of at least one of the first reflected lines corresponds to a light intensity of at least one of the first projected lines, or (2) a light intensity of at least one of the second reflected lines corresponds to a light intensity of at least one of the second projected lines.

9 . The system of claim 3 , wherein the projected junction characteristics and the reflected junction characteristics each comprises a distance between lines.

10 . The system of claim 9 , wherein at least one of the projected junction characteristics matches at least one of the reflected junction characteristics when:
(1) a distance between the first reflected line and another one of the first reflected lines corresponds to a distance between the first projected line and another one of the first projected lines that corresponds to the other first reflected line, or (2) a distance between the second reflected line and another one of the second reflected lines corresponds to a distance between the second projected line and another one of the second projected lines that corresponds to the other second reflected line.

11 . The system of claim 3 , wherein:
the reflected junction characteristics comprise one or more first reflected line characteristics associated with the one of the first reflected lines, one or more of the first projected lighting characteristics are associated with the one of the first projected lines, and at least one of the projected junction characteristics matches at least one of the reflected junction characteristics when one or more of the first projected lighting characteristics associated with the one of the first projected lines match one or more of the first reflected line characteristics.

12 . The system of claim 3 , wherein:
the reflected junction characteristics comprise one or more second reflected line characteristics associated with the one of the second reflected lines, one or more of the second projected lighting characteristics are associated with the second projected line, and at least one of the projected junction characteristics matches at least one of the reflected junction characteristics when one or more of the second projected lighting characteristics associated with the second projected line match one or more of the second reflected line characteristics.

13 . The system of claim 3 , wherein the processors are further configured to:
detect movement of the device or of the object; in response to detecting the movement, identify a new location of the reflected junction, wherein the new location of the reflected junction is on or near an epipolar line that intersects a previous location of the reflected junction; and determine, using triangulation, a depth associated with the new location of the reflected junction based on the camera, at least one of the projectors, a location of the projected junction and the new location of the reflected junction.

14 . The system of claim 13 , wherein the reflected junction is on or near the epipolar line when the reflected junction is located within a threshold distance of the epipolar line.

15 . The system of claim 3 , wherein the processors are further configured to:
propagate the reflected junction characteristics to one or more reflected junctions that are adjacent to the reflected junction.

16 . The system of claim 3 , wherein a confidence value is associated with the identified correspondence between the projected junction and the reflected ju

Assignees

• Facebook Tech Llc

Inventors

• Grossinger Nadav
• Garcia Ricardo
• Hasbun Robert
• Kantor Niv
• Romano Nitay

Classifications

• G06T2207/10016

  Video; Image sequence · CPC title

• G06T2207/10152

  Varying illumination · CPC title

• G06T7/521 Primary

  from laser ranging, e.g. using interferometry; from the projection of structured light · CPC title

• G06T7/579

  from motion · CPC title

• G06T7/20

  Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51) · CPC title