Methods and systems for capturing 3D surface geometry

Patent metadata

Field	Value
Publication number	US-10097813-B2
Application number	US-201213408488-A
Country	US
Kind code	B2
Filing date	Feb 29, 2012
Priority date	May 9, 2011
Publication date	Oct 9, 2018
Grant date	Oct 9, 2018

Abstract

A system for capturing 3D image data of a scene, including three light sources, each configured to emit light at a different wavelength to the other two sources and to illuminate the scene to be captured; a first video camera configured to receive light from the light sources which has been reflected from the scene, to isolate light received from each of the light sources, and to output data relating to the image captured for each of the three light sources; a depth sensor configured to capture depth map data of the scene; and an analysis unit configured to receive data from the first video camera and process the data to obtain data relating to a normal field obtained from the images captured for each of the three light sources, and to combine the normal field data with the depth map data to capture 3D image data of the scene.

First claim

The invention claimed is:

1. A system for capturing 3D video data of a dynamic scene, the video data comprising a plurality of frames, the system comprising:
three light sources, each light source configured to emit light at a different wavelength to the other two sources, the three light sources being configured to illuminate the scene to be captured;
a first video camera configured to receive light from the light sources which has been reflected from the scene, the first video camera being configured to perform a photometric stereo measurement by isolating light received from each of the light sources and output photometric stereo data relating to the image captured for each of the three light sources;
a depth sensor configured to obtain, using a non-photometric stereo measurement, non-photometric stereo data in the form of a first depth map of the scene, wherein the depth sensor will produce a depth map with lower frequency resolution in 2D Fourier space than the photometric stereo data;
a processor configured to:
receive said photometric stereo data from the first video camera and process said photometric stereo data to obtain data relating to a first normal field obtained from the images captured for each of the three light sources;
receive said non-photometric stereo data in the form of the first depth map of the scene obtained by the depth sensor using a non-photometric stereo measurement; and
combine the first normal field with that of the non-photometric data in the form of the first depth map by:
blurring the data relating to the first normal field, using a blurring kernel and wherein the variance of the blurring kernel is σ 2 where σ 2 is set at a value to reduce flickering of surfaces in the dynamic scene between subsequent frames;
calculating the rotation of the normals of the first normal field caused due to blurring;
calculating a further normal field from the non-photometric data that has been blurred by the same kernel; and
applying the calculated rotation of the normals of the first normal field caused due to blurring to the normals of the further normal field, to produce 3D image data of the scene.

2. A system for capturing 3D video data of a dynamic scene, the video data comprising a plurality of frames, the system comprising:
three light sources, each light source configured to emit light at a different wavelength to the other two sources, the three light sources being configured to illuminate the scene to be captured;
a first video camera configured to receive from the light sources light which has been reflected from the scene, the first video camera being configured to perform a photometric stereo measurement by isolating light received from each of the light sources and output photometric stereo data relating to the image captured for each of the three light sources;
a second video camera spatially separate from said first video camera, the first video camera and the second video camera being arranged in a two view stereo arrangement to perform a non-photometric stereo measurement to obtain non-photometric stereo data from the first and second video cameras, the non-photometric stereo data having a lower frequency resolution in 2D Fourier space than the photometric stereo data;
a processor configured to:
receive said photometric stereo data from the first video camera and process said photometric stereo data to obtain data relating to a first normal field obtained from the images captured for each of the three light sources;
receive non-photometric stereo data from the two-view stereo arrangement of the first and second video cameras,
produce data in the form of a depth map from the non-photometric stereo data, and
combine the first normal field data with that of the non-photometric data in the form of the depth map by:
blurring the data relating to the first normal field, using a blurring kernel and wherein the variance of the blurring kernel is σ 2 where σ 2 is set at a value to reduce flickering of surfaces in the dynamic scene between subsequent frames;
calculating the rotation of the normals of the first normal field caused due to blurring;
calculating a further normal field from the non-photometric data that has been blurred by the same kernel; and
applying the calculated rotation of the normals of the first normal field caused due to blurring to the normals of the further normal field, to produce 3D image data of the scene.

3. A system according to claim 1 , wherein said depth sensor comprises a projector configured to project a pattern on said scene and a camera configured to capture an image of said pattern projected onto the scene, said processor being configured to generate a second depth map from said captured image.

4. A system according to claim 1 , wherein said depth sensor comprises a time of flight sensor.

5. A method for producing 3D video data of a dynamic scene the video data comprising a plurality of frames, the method comprising:
receiving photometric stereo data from a first video camera, said photometric stereo data comprising video data captured by said first camera of the scene illuminated from three different directions, the data being capable of isolation into the image data captured for each illumination direction;
receiving non-photometric stereo data in the form of a first depth map of the scene obtained by a depth sensor using a non-photometric stereo measurement, wherein the first depth map will have lower frequency resolution in 2D Fourier space than the photometric stereo data;
analyzing the photometric stereo data received from the first video camera and processing said photometric stereo data to obtain data relating to a first normal field; the method further comprising combining the first normal field data with that of the non-photometric data in the form of the first depth map by blurring the data relating to the first normal field, using a blurring kernel and wherein the variance of the blurring kernel is σ 2 where σ 2 is set at a value to reduce flickering of surfaces in the dynamic scene between subsequent frames;
calculating the rotation of the normals of the first normal field caused due to blurring;
calculating a further normal field from the non-photometric data that has been blurred by the same kernel; and
applying the calculated rotation of the normals of the first normal field caused due to blurring to the normals of the further normal field, to produce 3D image data of the scene.

6. A method according to claim 5 , wherein the data from the first video camera is analyzed assuming that the scene to be imaged reflects light in accordance with a Lambertian imaging model, wherein:
n =( VL ) −1 c,

where c=[c 0 c 1 c 2 ] T where c 0 , c 1 , c 2 are the intensities measured at a pixel in an image captured from illuminating the scene from each of the three directions respectively, n is a vector expressing a normal to a surface of the pixel, V is a matrix which represents the combined response of the surface and the camera to the illumination, and L is a matrix determining the direction of the three illuminating lights.

7. A method according to claim 6 , further comprising a calibration procedure, said calibration procedure being configured to determine M, where M=VL.

8. A method according to claim 6 , further comprising a calibration procedure, said calibration procedure being configured to determine V and L separately.

9. A method according to claim 8 , wherein V is determined for each pixel.

10. A method according to claim 9 , wherein a plurality of matrices V are determined for each scene during calibration such that there are N number of matrices V, where N is an integer from 1 to a value le

Assignees

• Stenger Bjorn
• Maki Atsuto
• Perbet Frank
• Woodford Oliver
• Cipolla Roberto
• Anderson Robert
• Toshiba Kk

Inventors

• Stenger Bjorn
• Maki Atsuto
• Perbet Frank
• Woodford Oliver
• Cipolla Roberto
• Anderson Robert

Classifications

• H04N23/90

  Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums · CPC title

• H04N13/254 Primary

  in combination with electromagnetic radiation sources for illuminating objects · CPC title

• H04N13/25 Primary

  using two or more image sensors with different characteristics other than in their location or field of view, e.g. having different resolutions or colour pickup characteristics; using image signals from one sensor to control the characteristics of another sensor · CPC title

• G06T2207/10021

  Stereoscopic video; Stereoscopic image sequence · CPC title

• G06T2207/10152

  Varying illumination · CPC title