Methods and apparatus for enhancing depth maps with polarization cues

Patent metadata

Field	Value
Publication number	US-10260866-B2
Application number	US-201514979433-A
Country	US
Kind code	B2
Filing date	Dec 27, 2015
Priority date	Mar 6, 2015
Publication date	Apr 16, 2019
Grant date	Apr 16, 2019

Abstract

A 3D imaging system uses a depth sensor to produce a coarse depth map, and then uses the coarse depth map as a constraint in order to correct ambiguous surface normals computed from polarization cues. The imaging system outputs an enhanced depth map that has a greater depth resolution than the coarse depth map. The enhanced depth map is also much more accurate than could be obtained from the depth sensor alone. In many cases, the imaging system extracts the polarization cues from three polarized images. Thus, in many implementations, the system takes only three extra images—in addition to data used to generate the coarse depth map—in order to dramatically enhance the coarse depth map.

First claim

What is claimed is:

1. A method comprising:
(a) capturing a first set of images of an object, which first set of images comprises one or more images;
(b) capturing a second set of images of the object, which second set includes a first polarized image, a second polarized image and a third polarized image, in such a way that the first, second and third polarized images capture light that has been filtered at a first polarizer angle, a second polarizer angle and a third polarizer angle, respectively, the first, second and third polarizer angles being different from each other;
(c) calculating, based on the first set of images, a first depth map of a surface of the object;
(d) calculating, based on the second set of images, polarization surface normals of the surface; and
(e) performing a computation that (i) is based on the first depth map and the polarization surface normals, (ii) disregards gradients in a first region of the surface that has a lower degree of polarization than the rest of the surface, and (iii) calculates a second depth map of the surface, which second depth map is more accurate than the first depth map in part because the gradients in the first region of the surface are disregarded.

2. The method of claim 1 , wherein the calculating of the polarization surface normals takes into account refractive indices.

3. The method of claim 1 , wherein the calculating of the polarization surface normals takes into account behavior of light moving between media of different refractive indices.

4. The method of claim 1 , wherein the first and second set of images are not identical sets.

5. The method of claim 1 , wherein, in the calculating of the second depth map, the first depth map is a constraint on polarization normal data regarding the surface.

6. Apparatus comprising
(a) one or more cameras configured
(i) to capture a first set of images of an object, which first set of images comprises one or more images, and
(ii) to capture a second set of images of the object, which second set includes a first polarized image, a second polarized image and a third polarized image, such that the first, second and third polarized images capture light that has been filtered at a first polarizer angle, a second polarizer angle and a third polarizer angle, respectively, the first, second and third polarizer angles being different from each other; and

(b) one or more computers that are programmed
(i) to calculate, based on the first set of images, a first depth map of a surface of the object,
(ii) to calculate, based on the second set of images, polarization surface normals of the surface, and
(iii) to perform a computation that (A) is based on the first depth map and the polarization surface normals, (B) disregards gradients in a first region of the surface that has a lower degree of polarization than the rest of the surface, and (C) calculates a second depth map of the surface, which second depth map is more accurate than the first depth map in part because the gradients in the first region of the surface are disregarded.

7. The apparatus of claim 6 , wherein the one or more computers are programmed to calculate the polarization surface normals in a manner that takes into account refractive indices.

8. The apparatus of claim 6 , wherein the one or more computers are programmed to calculate the polarization surface normals in a manner that takes into account behavior of light moving between media of different refractive indices.

9. The apparatus of claim 6 , wherein the one or more computers are programmed to calculate the second depth map in such a way that the first depth map is a constraint on polarization normal data regarding the surface.

Assignees

• Massachusetts Inst Technology

Inventors

• Kadambi Achuta
• Taamazyan Vage
• Shi Boxin
• Raskar Ramesh

Classifications

• G06T7/514

  from specularities · CPC title

• G06T7/507

  from shading (G06T7/586 takes precedence) · CPC title

• G01B11/24 Primary

  for measuring contours or curvatures · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• G06T2207/10141

  Special mode during image acquisition · CPC title