Estimating surface properties using a plenoptic camera

Patent metadata

Field	Value
Publication number	US-9797716-B2
Application number	US-201514594017-A
Country	US
Kind code	B2
Filing date	Jan 9, 2015
Priority date	Jan 9, 2015
Publication date	Oct 24, 2017
Grant date	Oct 24, 2017

Abstract

A plenoptic camera captures a plenoptic image of an object illuminated by a point source (preferably, collimated illumination). The plenoptic image is a sampling of the four-dimensional light field reflected from the object. The plenoptic image is made up of superpixels, each of which is made up of subpixels. Each superpixel captures light from a certain region of the object (i.e., a range of x,y spatial locations) and the subpixels within a superpixel capture light propagating within a certain range of directions (i.e., a range of u,v spatial directions). Accordingly, optical properties estimation, surface normal reconstruction, depth estimation, and three-dimensional rendering can be provided by processing only a single plenoptic image. In one approach, the plenoptic image is used to estimate the bidirectional reflectance distribution function (BRDF) of the object surface.

First claim

What is claimed is:

1. A method implemented on a computer system for estimating surface properties of an object, the method comprising:
accessing a plenoptic image of the object illuminated by a point source, the plenoptic image comprising a plurality of superpixels, each superpixel capturing light reflected from a certain region of the object, and each superpixel comprising a plurality of subpixels, each subpixel capturing light reflected over a certain range of propagation directions; and
for a plurality of superpixels, processing the subpixels within the superpixel to estimate the surface properties for the region of the object corresponding to the superpixel, said processing based on the ranges of propagation directions captured by the subpixels.

2. The computer-implemented method of claim 1 wherein the surface properties include at least one of: surface normals for the different regions of the object, specular reflections for the different regions of the object, and surface roughnesses for the different regions of the object.

3. The computer-implemented method of claim 1 wherein processing the subpixels comprises processing the subpixels to estimate a bidirectional reflectance distribution function (BRDF) of the object.

4. The computer-implemented method of claim 1 wherein processing the subpixels comprises processing the subpixels using a mapping of regions and propagation directions from the object to corresponding subpixels.

5. The computer-implemented method of claim 4 wherein the mapping of regions and propagation directions from the object to corresponding subpixels depends on a geometry of the illumination by the point source, and both the geometry of the illumination and the mapping from the object to the subpixels are known before processing the subpixels.

6. The computer-implemented method of claim 4 further comprising:
calibrating a plenoptic imaging system that captures the plenoptic image, to determine the mapping from the object to the subpixels.

7. The computer-implemented method of claim 1 wherein the point source is a collimated source.

8. The computer-implemented method of claim 1 further comprising:
estimating a depth to a reference surface of the object, wherein estimating the surface properties of the object comprises estimating the surface properties of the object relative to the reference surface.

9. The computer-implemented method of claim 8 wherein estimating the depth to the reference surface of the object is based on an optical design of the plenoptic imaging system that captures the plenoptic image.

10. The computer-implemented method of claim 8 wherein estimating the depth to the reference surface of the object is based on processing the captured plenoptic image.

11. The computer-implemented method of claim 8 further comprising:
refocusing the plenoptic imaging system that captures the plenoptic image, to the estimated depth; and
capturing the plenoptic image of the object with the plenoptic imaging system refocused to the estimated depth.

12. The computer-implemented method of claim 1 wherein processing the subpixels comprises processing the subpixels from a single plenoptic image to estimate the surface properties of the object.

13. The computer-implemented method of claim 1 wherein:
accessing a plenoptic image comprises accessing a plurality of plenoptic images of the object illuminated by different point source geometries; and
processing the subpixels comprises processing the subpixels from the plurality of plenoptic images to estimate the surface properties of the object.

14. The computer-implemented method of claim 13 wherein the plurality of plenoptic images includes at least three plenoptic images and the surface properties include surface normals, specular reflections, surface roughnesses and diffuse reflections for the different regions of the object.

15. The computer-implemented method of claim 13 wherein the plurality of plenoptic images are spectrally coded and the plurality of plenoptic images are captured simultaneously.

16. The computer-implemented method of claim 1 further comprising:
generating a height map of the object from the plenoptic image and/or the estimated surface properties.

17. The computer-implemented method of claim 1 further comprising:
generating a three-dimensional rendering of the object from the plenoptic image and/or the estimated surface properties.

18. The computer-implemented method of claim 1 wherein:
accessing a plenoptic image comprises accessing a video with frames of plenoptic images captured sequentially in time; and
processing the subpixels comprises processing the subpixels to estimate surface properties for each frame of the video.

19. A system for estimating surface properties of an object, the system comprising:
a plenoptic imaging system that captures a plenoptic image of the object illuminated by a point source, the plenoptic image comprising a plurality of superpixels, each superpixel capturing light reflected from a certain region of the object, and each superpixel comprising a plurality of subpixels, each subpixel capturing light reflected over a certain range of propagation directions; and
a processor that, for a plurality of superpixels, processes the subpixels within the superpixel to estimate the surface properties for the region of the object corresponding to the superpixel, said processing based on the ranges of propagation directions captured by the subpixels.

20. A computer program product embedded in a non-transitory computer-readable medium for estimating surface properties of an object, the computer program product storing computer program code for performing a method, the method comprising:
accessing a plenoptic image of the object illuminated by a point source, the plenoptic image comprising a plurality of superpixels, each superpixel capturing light reflected from a certain region of the object, and each superpixel comprising a plurality of subpixels, each subpixel capturing light reflected over a certain range of propagation directions; and
for a plurality of superpixels, processing the subpixels within the superpixel to estimate the surface properties for the region of the object corresponding to the superpixel, said processing based on the ranges of propagation directions captured by the subpixels.

Assignees

• Meng Lingfei
• Lu Liyang
• Berkner Kathrin
• Tosic Ivana
• Ricoh Co Ltd

Inventors

• Meng Lingfei
• Lu Liyang
• Berkner Kathrin
• Tosic Ivana

Classifications

• H04N23/56 Primary

  provided with illuminating means · CPC title

• G01B11/25 Primary

  by projecting a pattern, e.g. {one or more lines,} moiré fringes on the object (G01B11/255 takes precedence {; image analysis for depth or shape recovery G06T7/50}) · CPC title

• H04N13/0232

  Electricity · mapped topic

• G01B11/303

  using photoelectric detection means · CPC title

• G01B11/24

  for measuring contours or curvatures · CPC title