Augmenting a depth map representation with a reflectivity map representation

Patent metadata

Field	Value
Publication number	US-2016335773-A1
Application number	US-201615153741-A
Country	US
Kind code	A1
Filing date	May 13, 2016
Priority date	May 13, 2015
Publication date	Nov 17, 2016
Grant date	—

Abstract

A solution for generating a 3D representation of an object in a scene is provided. A depth map representation of the object is combined with a reflectivity map representation of the object to generate the 3D representation of the object. The 3D representation of the object provides more complete and accurate information of the object. An image of the object is illuminated by structured light and is captured. Pattern features rendered in the captured image of the object are analyzed to derive a depth map representation and a reflectivity map representation of the illuminated object. The depth map representation provides depth information while the reflectivity map representation provides surface information (e.g., reflectivity) of the illuminated object. The 3D representation of the object can be enhanced with additional illumination projected onto the object and additional images of the object.

First claim

What is claimed is:

1 . A method comprising:
illuminating an object in a scene with structured light of an illumination pattern; capturing an image of the illuminated object, the image of the illuminated object having a plurality of pattern features according to the illumination pattern; analyzing at least one pattern feature in the captured image of the illuminated object; generating a depth map representation of the object based on the analysis of the at least one pattern feature of the captured image, the depth map representation of the object comprising information describing distance of the object relative to a viewpoint when capturing the image of the object; generating a reflectivity map representation from the captured image of the object, the reflective map representation of the object comprising information describing reflectivity characteristics of at least a portion of the object; combining the depth map representation of the object and the reflectivity map representation of the object; and generating a three dimensional (3D) representation of the object based on the combination of the depth map representation of the object and the reflectivity map representation of the object.

2 . The method of claim 1 , wherein generating the reflectivity map representation of the object, comprises:
indexing the at least one pattern feature projected onto the object in the captured image; deriving functional behavior of the at least one indexed pattern feature; and removing the at least one pattern feature from the captured image of the object based on the derived functional behavior of the pattern feature.

3 . The method of claim 2 , wherein indexing the at least one pattern feature comprises:
labeling the at least one pattern feature based on feature characteristics of the pattern feature, the feature characteristics including at least one of intensity, shape, size or color of the pattern feature.

4 . The method of claim 2 , wherein the functional behavior of the at least one pattern feature is represented by intensity of the pattern feature along a predetermined direction.

5 . The method of claim 2 , wherein removing the at least one pattern feature comprises:
applying an multiplication function to the at least one pattern feature; and removing the at least one pattern feature based on the application of the multiplication function.

6 . The method of claim 1 , wherein removing the at least one pattern feature comprises:
determining positions of the at least one pattern feature with respect to centers of pixels of the object in the captured image; and correcting the positions of the at least one pattern feature with respect to the centers of the pixels of the object, wherein the at least one pattern feature is removed from the captured image of the object after the correction.

7 . The method of claim 1 , further comprising:
providing additional illumination to the object; and improving the reflectivity map representation of the object with the additional illumination to the object, the improved reflectivity map representation of the object providing additional information describing the reflectivity of one or more portions of the surface of the object.

8 . The method of claim 1 , further comprising generating an enhanced 3D representation of the object, wherein generating the enhanced 3D representation of the object comprises:
capturing one or more additional images of the object; registering the one or more additional images onto the depth map representation of the object with the reflectivity map representation; deriving additional 3D data from the additional image; and generating an enhanced 3D representation of the object with the additional 3D data.

9 . The method of claim 8 , further comprising:
generating the enhanced 3D representation of the object with an improved reflectivity map representation of the object.

10 . The method of claim 1 , wherein the pattern features are stripes with different intensities, adjacent two stripes being separated by a dark interval.

11 . A non-transitory computer readable storage medium storing computer program instructions, the computer program instructions when executed by a computer processor causes the processor to perform the steps of:
illuminating an object in a scene with structured light of an illumination pattern; capturing an image of the illuminated object, the image of the illuminated object having a plurality of pattern features according to the illumination pattern; analyzing at least one pattern feature in the captured image of the illuminated object; generating a depth map representation of the object based on the analysis of the at least one pattern feature of the captured image, the depth map representation of the object comprising information describing distance of the object relative to a viewpoint when capturing the image of the object; generating a reflectivity map representation from the captured image of the object, the reflective map representation of the object comprising information describing reflectivity characteristics of at least a portion of the object; combining the depth map representation of the object and the reflectivity map representation of the object; and generating a three dimensional (3D) representation of the object based on the combination of the depth map representation of the object and the reflectivity map representation of the object.

12 . The non-transitory computer readable storage medium of claim 11 , wherein generating the reflectivity map representation of the object, comprises:
indexing the at least one pattern feature projected onto the object in the captured image; deriving functional behavior of the at least one indexed pattern feature; and removing the at least one pattern feature from the captured image of the object based on the derived functional behavior of the pattern feature.

13 . The non-transitory computer readable storage medium of claim 12 , wherein indexing the at least one pattern feature comprises:
labeling the at least one pattern feature based on feature characteristics of the pattern feature, the feature characteristics including at least one of intensity, shape, size or color of the pattern feature.

14 . The non-transitory computer readable storage medium of claim 12 , wherein the functional behavior of the at least one pattern feature is represented by intensity of the pattern feature along a predetermined direction.

15 . The non-transitory computer readable storage medium of claim 12 , wherein removing the at least one pattern feature comprises:
applying an multiplication function to the at least one pattern feature; and removing the at least one pattern feature based on the application of the multiplication function.

16 . The non-transitory computer readable storage medium of claim 11 , wherein removing the at least one pattern feature comprises:
determining positions of the at least one pattern feature with respect to centers of pixels of the object in the captured image; and correcting the positions of the at least one pattern feature with respect to the centers of the pixels of the object, wherein the at least one pattern feature is removed from the captured image of the object after the correction.

17 . The non-transitory computer readable storage medium of claim 11 , further comprising:
providing additional illumination to the object; and improving the reflectivity map representation of the object with the additional illumination to the object, the improved reflecti

Assignees

• Oculus Vr Llc

Inventors

• Romano Nitay
• Grossinger Nadav

Classifications

• G02B30/23

  using wavelength separation, e.g. using anaglyph techniques · CPC title

• G02B27/0149

  characterised by mechanical features (G02B27/0176 takes precedence) · CPC title

• G06T2207/10024

  Color image · CPC title

• G06T2207/30201

  Face · CPC title

• G02B2027/0138

  comprising image capture systems, e.g. camera · CPC title