Laser enhanced reconstruction of 3D surface

Patent metadata

Field	Value
Publication number	US-11022433-B2
Application number	US-201113577456-A
Country	US
Kind code	B2
Filing date	Jan 14, 2011
Priority date	Feb 12, 2010
Publication date	Jun 1, 2021
Grant date	Jun 1, 2021

Abstract

A method for reconstructing a surface of a three-dimensional object involves a projection of a laser spot pattern onto the surface of the three-dimensional object by a laser, and a generation of a series of endoscopic images as an endoscope is translated and/or rotated relative to the three-dimensional object. Each endoscopic image illustrates a different view of a laser spot array within the laser spot pattern as projected onto the surface of the three-dimensional object by the laser. The laser spot array may be identical to or a subset of the laser spot pattern. The method further involves a reconstruction of the surface of the three-dimensional object from a correspondence of the different views of the laser spot array as illustrated in the endoscopic images.

First claim

The invention claimed is:

1. A system comprising:
a laser configured to project a spot array defined within a projected spot pattern onto corresponding surface portions of a surface of a three-dimensional object;
an endoscope configured to be translated and/or rotated relative to the three-dimensional object to generate a plurality of 2D images each illustrating at least one of a different viewing angle and direction relative to the projected spot array within the projected spot pattern as projected onto the corresponding surface portions of the three-dimensional object by the laser and being further configured to ensure that the entire projected spot array projected on the corresponding surface portions of the three-dimensional object is visible in each of the plurality of 2D images and configured to ensure that a distance between the endoscope and the three-dimensional object changes between each of the plurality of 2D images relative to the corresponding surface portions of the three-dimensional object that the projected spot array is projected onto; and
an image processing device in communication with the endoscope and configured to reconstruct a 3D image of the surface of the three-dimensional object from a correspondence of differing views of the spot array between each of the generated plurality of 2D images, wherein the image processing device is configured to reconstruct the 3D image of the surface of the three-dimensional object using coordinates that only include 2D coordinates from each of the generated plurality of 2D images that are each captured at different distances between the endoscope and the three-dimensional object.

2. The system of claim 1 , wherein the projected spot array includes all of the projected spot pattern.

3. The system of claim 1 , wherein the projected spot array on the corresponding surface portions of the three-dimensional object is a subset of the projected spot pattern.

4. The system of claim 1 , wherein the device is further configured to
generate a fundamental matrix for relating the different viewing angles and/or directions of the projected spot array on the corresponding surface portions of the three-dimensional object in the generated plurality of 2D images; and
reconstruct points on the three-dimensional object as a function of the fundamental matrix and the correspondence between the different viewing angles and/or directions of the projected spot array on the corresponding surface portions of the three-dimensional object.

5. The system of claim 1 , wherein the device is further configured to
generate a fundamental matrix for relating the different viewing angles and/or directions of the projected spot array on the corresponding surface portions of the three-dimensional object in the generated plurality of 2D images;
detect surface features of the object from the generated plurality of 2D images; and
reconstruct points on the three-dimensional object as a function of the fundamental matrix and the surface features of the object detected in the generated plurality of 2D images.

6. The system of claim 1 , wherein the device is further configured to generate a fundamental matrix for relating the different viewing angles and/or directions of the projected spot array on the corresponding surface portions of the three-dimensional object in the generated plurality of 2D images.

7. The system of claim 6 , wherein the endoscope includes a camera, and the device is further configured to generate an essential matrix for relating the different viewing angles and/or directions of the projected spot array on the corresponding surface portions of the three-dimensional object in the generated plurality of 2D images, the essential matrix being a function of the fundamental matrix and a calibration matrix associated with calibration of the camera.

8. The system of claim 7 , wherein the device is further configured to generate a translation vector and a rotation matrix as a function of the essential matrix.

9. The system of claim 8 , wherein the device is further configured to generate a projection matrix for each of the different viewing angles and/or directions of the projected spot array on the corresponding surface portions of the three-dimensional object in the generated plurality of 2D images as a function of the translation vector and the rotation matrix, each projection matrix being a linear transformation of an associated view of the projected spot array in one of the generated plurality of 2D images.

10. The system of claim 9 , wherein the device is further configured to reconstruct points on the three-dimensional object as a function of each projection matrix and the associated different viewing angles and/or directions of the projected spot array on the corresponding surface portions of the three-dimensional object from the generated plurality of 2D images.

11. The system of claim 9 , wherein the device is further configured to
detect surface features of the object in the generated plurality of 2D images for each view of the different viewing angles and/or directions of the projected spot array on the corresponding surface portions of the three-dimensional object; and
reconstruct points on the three-dimensional object as a function of each projection matrix and each surface feature of the object detected in the different viewing angles and/or directions in the generated plurality of 2D images.

12. The system of claim 1 , wherein the endoscope is intra-operatively calibrated from at least two of the plurality of 2D images.

13. A system comprising:
an endoscope configured to be translated and/or rotated relative to a three-dimensional object to generate a plurality of 2D images each illustrating at least one of a different viewing angle and direction of a spot array included within a spot pattern as projected onto corresponding surface portions of the three-dimensional object and being further configured to ensure that the entire projected spot array projected on the corresponding surface portions of the three-dimensional object is visible in each of the plurality of 2D images and configured to ensure that a distance between the endoscope and the three-dimensional object changes between each of the plurality of 2D images relative to the corresponding surface portions of the three-dimensional object that the spot array is projected onto; and
an image processing device in communication with the endoscope and configured to reconstruct a 3D image of the surface of the three-dimensional object from a correspondence of differing views of the spot array between each of the plurality of 2D images, wherein the image processing device is configured to reconstruct the 3D image of the surface of the three-dimensional object using coordinates that only include 2D coordinates from each of the generated plurality of 2D images that are each captured at different distances between the endoscope and the three-dimensional object.

14. The system of claim 13 , wherein the device is further configured to
generate a fundamental matrix for relating the different viewing angles and/or directions of the projected spot array on the corresponding surface portions of the three-dimensional object in the generated plurality of 2D images; and
reconstruct points on the three-dimensional object as a function of the fundamental matrix and the different viewing angles and/or directions of the projected spot array on the corresponding surface portions of the three-dimensional object.

15. The system of claim 13 , wherein the device is further configured to
generate a fundamental matrix for relating the different

Assignees

• Popovic Aleksandra
• Koninklijke Philips Nv

Inventors

• Popovic Aleksandra

Classifications

• A61B1/0605

  for spatially modulated illumination · CPC title

• A61B1/04

  combined with photographic or television appliances · CPC title

• A61B1/00009

  of image signals during a use of endoscope · CPC title

• G06T7/97

  Determining parameters from multiple pictures (depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85) · CPC title

• A61B5/1077

  Measuring of profiles · CPC title