Method and apparatus for determining tumor shift during surgery using a stereo-optical three-dimensional surface-mapping system

Patent metadata

Field	Value
Publication number	US-9336592-B2
Application number	US-201314375311-A
Country	US
Kind code	B2
Filing date	Feb 1, 2013
Priority date	Feb 3, 2012
Publication date	May 10, 2016
Grant date	May 10, 2016

Abstract

A system and method for determining intraoperative locations of a lesion in tissue from lesion locations determined in preoperative imaging includes determining three dimensional locations of surface features of the organ in the preoperative images. A preoperative surface map is extracted from stereo images annotated with surface features from preoperative images. An intraoperative surface map of the organ is extracted from stereo images, and surface features are identified in the stereo images corresponding to surface features annotated into the preoperative surface map. Three dimensional displacements of the surface features are determined and used to constrain a computer model of deformation of the organ. In embodiments, the model of deformation is adapted or constrained to model locations and dimensions of surgical cavities using an optical flow method and/or locations of surgical instruments in the organ. The model of deformation is used to determine intraoperative locations for the lesion.

First claim

What is claimed is:

1. A method for determining a 3D model of a surface comprising:
calibrating 3D reconstruction parameters for at least one reference setting of an optical system;
calibrating image warping parameters for at least one secondary calibration settings, the image warping parameters adapted to control an image warping routine to warp images taken at that secondary calibration setting into warped images corresponding to images taken at the reference setting;
taking an observed stereo image through the optical system with the optical system at a current setting;
determining warping parameters from the image warping parameters for at least one secondary calibration setting of the at least one secondary calibration settings, the warping parameters for warping the observed stereo image taken through the optical system with the optical system at the current setting into a warped stereo image corresponding to a stereo image taken at the reference setting;
warping the observed stereo image at the current setting into the warped stereo image corresponding to a stereo image taken at the reference setting;
determining three-dimensional (3D) warping parameters for warping a first image of the warped stereo image corresponding to a stereo image taken at the reference setting into a second image of the warped stereo image corresponding to a stereo image taken at the reference setting; and
using the 3D warping parameters to determine the 3D model of the surface.

2. The method of claim 1 wherein there are image warping parameters for more than one secondary calibration setting stored in a table, and wherein the step of determining warping parameters from the image warping parameters for at least one secondary calibration setting includes interpolating between image warping parameters stored in the table.

3. The method of claim 2 wherein the step of determining 3D warping parameters uses vertically unconstrained fitting, and wherein pixels initially having vertical warp parameters exceeding a threshold are excluded from fitting the 3D warping parameters.

4. The method of claim 3 further comprising using the 3D model of the surface to constrain a mechanical model of tissue to determine a displacement of a particular structure in the tissue.

5. The method of claim 4 further comprising extracting a 3 dimensional representation of a surgical cavity, and wherein the mechanical model of tissue is adapted with the 3-dimensional model of the surgical cavity such that the cavity is taken into account in determining displacement of the particular structure in the tissue; wherein the 3-dimensional representation of a surgical cavity is determined by projecting a pre-resection 3D model of the surface onto a plane as a first projection, projecting a post-resection model of the surface onto the same plane as a second projection, determining an image-warping that maps the first projection into the second projection or the second projection into the first projection, this mapping is used with the pre-resection and post-resection models of the surface to determine a 3-dimensional model of difference, and the 3-dimensional representation of the surgical cavity is derived from the 3-dimensional model of difference.

6. The method of claim 4 further comprising locating a surgical tool, and wherein the mechanical model of tissue is adapted with a location of the surgical tool such that presence and location of the tool is taken into account in determining displacement of the particular structure in the tissue.

7. The method of claim 1 wherein the step of determining 3D warping parameters uses vertically unconstrained fitting, and wherein pixels initially having vertical warp parameters exceeding a threshold are excluded from fitting the 3D warping parameters.

8. The method of claim 7 further comprising using the 3D model of the surface to constrain a mechanical model of tissue to determine a displacement of a particular structure in the tissue.

9. The method of claim 8 further comprising extracting a 3 dimensional representation of a surgical cavity, and wherein the mechanical model of tissue is adapted with the 3-dimensional model of the surgical cavity such that the cavity is taken into account in determining displacement of the particular structure in the tissue; wherein the 3-dimensional representation of a surgical cavity is determined by projecting a pre-resection 3D model of the surface onto a plane as a first projection, projecting a post-resection model of the surface onto the same plane as a second projection, determining an image-warping that maps the first projection into the second projection or the second projection into the first projection, this mapping is used with the pre-resection and post-resection models of the surface to determine a 3-dimensional model of difference, and the 3-dimensional representation of the surgical cavity is derived from the 3-dimensional model of difference.

10. The method of claim 9 further comprising locating a surgical tool, and wherein the mechanical model of tissue is adapted with a location of the surgical tool such that presence and location of the tool is taken into account in determining displacement of the particular structure in the tissue.

11. A system for determining a 3D model of a surface comprising:
an optical system having a plurality of settings, each setting providing a specific focal length and magnification, the optical system comprising an encoder for observing a current setting of the optical system;
a memory configured to contain calibrated 3D reconstruction parameters for at least one reference setting of the optical system;
the memory further configured with image warping parameters for at least one secondary calibration setting, the image warping parameters adapted to control an image warping routine to warp images taken at that secondary calibration setting into warped images corresponding to images taken at a reference setting of the at least one reference setting;
a camera coupled to capture an observed stereo image through the optical system at a current setting;
a processor configured with machine readable instructions in the memory, the machine readable instructions comprising instructions for determining warping parameters from the current setting and image warping parameters for at least one secondary calibration setting, the warping parameters being warping parameters adapted for warping the observed stereo image into a warped stereo image corresponding to a stereo image taken at the reference setting;
the memory further configured with machine readable instructions for warping the observed stereo image into the warped stereo image corresponding to a stereo image taken at the reference setting;
the memory further configured with machine readable instructions for determining three-dimensional (3D) warping parameters for warping a first image of the warped stereo image corresponding to a stereo image taken at the reference setting into a second image of the stereo image corresponding to a stereo image taken at the reference setting;
the memory further configured with machine readable instructions for using the 3D warping parameters for determining the 3D model of the surface.

12. The system of claim 11 wherein the memory is configured with image warping parameters for more than one secondary calibration setting stored in a table, and wherein the machine readable instructions for determining warping parameters from the image warping parameters for at least one secondary calibration setting includes instructions for interpolating between image warping parameters stored in the table.

13. The system o

Assignees

• Dartmouth College

Inventors

• Fan Xiaoyao
• Roberts David W
• Paulsen Keith D
• Ji Songbai
• Hartov Alex

Classifications

• G06T7/0012 Primary

  Biomedical image inspection · CPC title

• G06T17/00 Primary

  Three-dimensional [3D] modelling for computer graphics · CPC title

• A61B5/055

  involving electronic [EMR] or nuclear [NMR] magnetic resonance, e.g. magnetic resonance imaging · CPC title

• G06T2207/10021

  Stereoscopic video; Stereoscopic image sequence · CPC title

• G06T7/0028

  Physics · mapped topic