Methods for autoregistration of arthroscopic video images to preoperative models and devices thereof

Patent metadata

Field	Value
Publication number	US-12343218-B2
Application number	US-202117998835-A
Country	US
Kind code	B2
Filing date	Jun 16, 2021
Priority date	Jun 18, 2020
Publication date	Jul 1, 2025
Grant date	Jul 1, 2025

Abstract

Surgical methods and devices that facilitate registration of arthroscopic video to preoperative models are disclosed. With this technology, a machine learning model is applied to diagnostic video data captured via an arthroscope to identify an anatomical structure. An anatomical structure in a three-dimensional (3D) anatomical model is registered to the anatomical structure represented in the diagnostic video data. The 3D anatomical model is generated from preoperative image data. The anatomical structure is then tracked intraoperatively based on the registration and without requiring fixation of fiducial markers to the patient anatomy. A simulated projected view of the registered anatomical structure is generated from the 3D anatomical model based on a determined orientation of the arthroscope during capture of intraoperative video data. The simulated projected view is scaled and oriented based on one or more landmark features of the anatomical structure extracted from the intraoperative video data.

First claim

We claim:

1. A method for registration of arthroscopic video images to preoperative models, the method implemented by one or more surgical computing devices and comprising:
applying a machine learning model to diagnostic video data captured via an arthroscope to identify an anatomical structure represented in the diagnostic video data;
registering one of a plurality of anatomical structures in a three-dimensional (3D) anatomical model to the anatomical structure represented in the diagnostic video data, wherein the 3D anatomical model is generated from preoperative image data;
tracking the anatomical structure intraoperatively based on the registration;
generating a simulated projected view of the registered one of the plurality of anatomical structures from the 3D anatomical model based on a determined orientation of the arthroscope during capture by the arthroscope of intraoperative video data; and
outputting the simulated projected view scaled and oriented based on one or more landmark features of the anatomical structure extracted from the intraoperative video data.

2. The method of claim 1 , further comprising:
generating an overlay comprising the scaled and oriented simulated projected view;
merging the generated overlay with the intraoperative video data based on the registration to generate merged video data; and
outputting to a display device the merged video data.

3. The method of claim 2 , further comprising determining a stage of a surgical procedure based on an obtained surgical plan for the surgical procedure and identification of the anatomical structure in the intraoperative video data, wherein the generated overlay further comprises guidance extracted from the obtained surgical plan.

4. The method of claim 3 , wherein the guidance comprises one or more of:
textual directions associated with a current task in the surgical procedure; or
a visual indication of another one of the plurality of anatomical structures corresponding to a subsequent task in the surgical procedure.

5. The method of claim 2 , further comprising obtaining an annotated version of the 3D anatomical model or the preoperative image data that identifies an anatomical point corresponding to a portion of patient anatomy or at least one of the one or more landmark features, wherein the generated overlay further comprises an indication of the anatomical point.

6. The method of claim 2 , wherein the display device comprises a mixed reality headset and the method further comprises:
tracking one or more of a position or an orientation of the mixed reality headset; and
generating the overlay using a field of view of the arthroscope to determine a local reference frame and based on a known spatial and scale relationship between the arthroscope field of view and another reference frame of the mixed reality headset determined based on the tracking.

7. The method of claim 1 , further comprising training the machine learning model based on additional video data comprising a plurality of image frames each comprising at least one annotated representation of one or more of the plurality of anatomical structures.

8. The method of claim 1 , further comprising:
determining an eye position of a user; and
outputting to a projector the scaled and oriented simulated projected view for projection by the projector onto patient skin based on the determined eye position.

9. The method of claim 1 , wherein the anatomical structure represented in the intraoperative video data comprises soft tissue and the method further comprises:
determining a size and position of a first portion of the soft tissue from the intraoperative video data; and
applying another machine learning model to the 3D anatomical model and the determined size and position to generate a representation of a second portion of the soft tissue in a morphed state, wherein the simulated projected view comprises the representation of the second portion of the soft tissue in the morphed state.

10. The method of claim 1 , further comprising:
generating a weighting value for each of a plurality of portions of the 3D anatomical model; and
generating the simulated projected view to include a subset of the plurality of portions based on the weighting values.

11. A surgical computing device, comprising:
a non-transitory computer-readable medium comprising programmed instructions stored thereon for registration of arthroscopic video images to preoperative models; and
one or more processors coupled to the non-transitory computer-readable medium and configured to execute the stored programmed instructions which causes the one or more processors to:
apply a machine learning model to diagnostic video data captured via an arthroscope to identify an anatomical structure represented in the diagnostic video data;
register one of a plurality of anatomical structures in a three-dimensional (3D) anatomical model to the anatomical structure represented in the diagnostic video data, wherein the 3D anatomical model is generated from preoperative image data;
track the anatomical structure intraoperatively based on the registration;
generate a simulated projected view of the registered one of the plurality of anatomical structures from the 3D anatomical model based on a determined orientation of the arthroscope during capture by the arthroscope of intraoperative video data; and
output the simulated projected view scaled and oriented based on one or more landmark features of the anatomical structure extracted from the intraoperative video data.

12. The surgical computing device of claim 11 , wherein the stored programmed instructions further cause the one or more processors to:
generate an overlay comprising the scaled and oriented simulated projected view;
merge the generated overlay with the intraoperative video data based on the registration to generate merged video data; and
output to a display device the merged video data.

13. The surgical computing device of claim 12 , wherein the stored programmed instructions further cause the one or more processors to determine a stage of a surgical procedure based on an obtained surgical plan for the surgical procedure and identification of the anatomical structure in the intraoperative video data, wherein the generated overlay further comprises guidance extracted from the obtained surgical plan.

14. The surgical computing device of claim 13 , wherein the guidance comprises one or more of:
textual directions associated with a current task in the surgical procedure; or
a visual indication of another one of the plurality of anatomical structures corresponding to a subsequent task in the surgical procedure.

15. The surgical computing device of claim 12 , wherein the stored programmed instructions further cause the one or more processors to obtain an annotated version of the 3D anatomical model or the preoperative image data that identifies an anatomical point corresponding to a portion of patient anatomy or at least one of the one or more landmark features, wherein the generated overlay further comprises an indication of the anatomical point.

16. The surgical computing device of claim 12 , wherein the display device comprises a mixed reality headset and the stored program instructions further cause the one or more processors to:
track one or more of a position or an orientation of the mixed reality headset; and
generate the overlay using a field of view of the arthroscope to determine a local reference frame and based on a known spatial and scale relationship between the arthroscope field of view and another reference frame

Assignees

• Smith & Nephew Inc
• Smith & Nephew Orthopaedics Ag
• Smith & Nephew Asia Pacific Pte Ltd

Inventors

• Quist Brian William
• Netravali Nathan Anil
• Torrie Paul Alexander

Classifications

• G06N3/09

  Supervised learning · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06T2207/30008

  Bone · CPC title

• A61B2090/368

  changing the image on a display according to the operator's position · CPC title

• A61B2090/366

  using projection of images directly onto the body · CPC title