Ultrasound bone registration with learning-based segmentation and sound speed calibration

Patent metadata

Field	Value
Publication number	US-11701090-B2
Application number	US-201815999152-A
Country	US
Kind code	B2
Filing date	Aug 16, 2018
Priority date	Aug 16, 2017
Publication date	Jul 18, 2023
Grant date	Jul 18, 2023

Abstract

A workflow is disclosed to accurately register ultrasound imaging to co-modality imaging. The ultrasound imaging is segmented with a convolutional neural network to detect a surface of the object. The ultrasound imaging is calibrated to reflect a variation in propagation speed of the ultrasound waves through the object by minimizing a cost function that sums the differences between the first and second steered frames, and compares the first and second steered frames of the ultrasound imaging with a third frame of the ultrasound imaging that is angled between the first and second steered frames. The ultrasound imaging is temporarily calibrated with respect to a tracking coordinate system by creating a point cloud of the surface and calculating a set of projection values of the point cloud to a vector. The ultrasound imaging, segmented and calibrated, is automatically registered to the co-modality imagine.

First claim

What is claimed is:

1. A method of operating a surgical system, the surgical system comprising an imaging device, a robotic manipulator configured to move a surgical instrument to manipulate an anatomical object with a working end of the surgical instrument, and one or more controllers, the method comprising:
generating, with the imaging device, ultrasound imaging of the anatomical object by propagating ultrasound waves along a plurality of scanlines through the anatomical object;
segmenting, with the one or more controllers, the ultrasound imaging with a convolutional neural network to detect a surface of the anatomical object by generating a probability map with the convolutional neural network and extracting the surface from the probability map for each of the plurality of scanlines;
calibrating, with the one or more controllers, the ultrasound imaging to reflect a variation in propagation speed of ultrasound waves through the anatomical object by comparing first and second steered frames of the ultrasound imaging with a third frame of the ultrasound imaging that is angled between the first and second steered frames, and wherein calibrating further comprises computing a first difference between the first steered frame and the third frame, computing a second difference between the second steered frame and the third frame, computing a sum of the first and second differences, and optimizing the sum to minimize a cost function;
temporally calibrating, with the one or more controllers, the ultrasound imaging with respect to a tracking coordinate system by creating a point cloud of the surface, calculating a set of projection values of the point cloud to a vector oriented relative to the plurality of scanlines, and calculating a temporal lag for the ultrasound imaging by minimizing variance of the set of projection values;
obtaining, with the one or more controllers, co-modality imaging of the anatomical object and a predefined boundary associated with the anatomical object;
automatically registering, with the one or more controllers, the segmented and calibrated ultrasound imaging to the co-modality imaging; and
after automatically registering, controlling, with the one or more controllers, the robotic manipulator for moving the surgical instrument to manipulate the anatomical object while preventing the working end of the surgical instrument from extending beyond the predefined boundary.

2. The method of claim 1 , wherein the vector is a three-dimensional vector oriented parallel to an average direction of the plurality of scanlines.

3. The method of claim 1 , wherein the step of calibrating the ultrasound imaging further comprises:
estimating the propagation speed of the ultrasound waves through the anatomical object based on the sum; and
calibrating the ultrasound imaging based on the estimated propagation speed.

4. The method of claim 1 , wherein the third frame is perpendicular to the anatomical object.

5. The method of claim 1 , wherein the co-modality imaging is a pre-operative computed tomography (CT) or magnetic resonance imaging (MRI) scan, and the ultrasound imaging is generated intraoperatively.

6. The method of claim 1 , wherein each of the following steps are performed during or immediately after generation of the ultrasound imaging:
segmenting the ultrasound imaging;
calibrating the ultrasound imaging to reflect the variation in propagation speed of the ultrasound waves through the anatomical object;
temporally calibrating the ultrasound imaging with respect to the tracking coordinate system; and
automatically registering the ultrasound imaging to the co-modality imaging.

7. The method of claim 1 , wherein the anatomical object is further defined as a bone, and wherein the predefined boundary defines a surface of the bone that should remain after a procedure.

8. The method of claim 1 , wherein a detectable marker is coupled to the anatomical object, the method further comprising:
tracking, with a tracking system, the anatomical object by detecting a position of the detectable marker in the tracking coordinate system.

9. The method of claim 1 , wherein a tracker is coupled to the imaging device, the method further comprising:
tracking, with a tracking system, the imaging device by detecting a position of the tracker in the tracking coordinate system.

10. The method of claim 1 , wherein each of the scanlines has a maximum gradient and a maximum intensity, wherein the surface is extracted as a center pixel between the maximum gradient and the maximum intensity along the scanlines.

11. A surgical system comprising:
an imaging device configured to generate ultrasound imaging of an anatomical object;
a robotic manipulator configured to move a surgical instrument to manipulate the anatomical object with a working end of the surgical instrument; and
one or more controllers coupled to the imaging device and the robotic manipulator, wherein the one or more controllers are configured to:
generate, with the imaging device, ultrasound imaging of the anatomical object by propagation of ultrasound waves along a plurality of scanlines through the anatomical object;
segment the ultrasound imaging with a convolutional neural network to detect a surface of the anatomical object by generation of a probability map with the convolutional neural network and extraction of the surface from the probability map for each of the plurality of scanlines;
calibrate the ultrasound imaging to reflect a variation in propagation speed of ultrasound waves through the anatomical object by comparison of first and second steered frames of the ultrasound imaging with a third frame of the ultrasound imaging that is angled between the first and second steered frames, and wherein to calibrate, the one or more controllers further compute a first difference between the first steered frame and the third frame, compute a second difference between the second steered frame and the third frame, compute a sum of the first and second differences, and optimize the sum to minimize a cost function;
temporally calibrate the ultrasound imaging with respect to a tracking coordinate system by creation of a point cloud of the surface, calculation of a set of projection values of the point cloud to a vector oriented relative to the plurality of scanlines, and calculation of a temporal lag for the ultrasound imaging by minimization of variance of the set of projection values;
obtain co-modality imaging of the anatomical object and a predefined boundary associated with the anatomical object;
automatically register the segmented and calibrated ultrasound imaging to the co-modality imaging; and
after automatic registration, control the robotic manipulator to move the surgical instrument to manipulate the anatomical object and prevent the working end of the surgical instrument from extending beyond the predefined boundary.

12. The surgical system of claim 11 , wherein the one or more controllers calibrate the ultrasound imaging by further being configured to:
estimate the propagation speed of the ultrasound waves through the anatomical object based on the sum; and
calibrate the ultrasound imaging based on the estimated propagation speed.

13. The surgical system of claim 11 , wherein the co-modality imaging is a pre-operative computed tomography (CT) or magnetic resonance imaging (MRI) scan, and the ultrasound imaging is generated intraoperatively.

14. The surgical system of claim 11 , wherein the anatomical object is further defined as a bone, and wherein the predefined boundary defines a surface of the bone that should remain after a procedure.

Assignees

• Mako Surgical Corp

Inventors

• Moctezuma De La Barrera José Luis
• Salehi Mehrdad
• Prevost Raphael
• Wein Wolfgang

Classifications

• G06T7/33 Primary

  using feature-based methods · CPC title

• G06T2207/10132

  Ultrasound image · CPC title

• A61B8/5207 Primary

  involving processing of raw data to produce diagnostic data, e.g. for generating an image · CPC title

• A61B6/5247

  combining images from an ionising-radiation diagnostic technique and a non-ionising radiation diagnostic technique, e.g. X-ray and ultrasound · CPC title

• A61B8/0875

  for diagnosis of bone · CPC title