Systems and methods for augmented reality display in navigated surgeries

Patent metadata

Field	Value
Publication number	US-2021121237-A1
Application number	US-201816494540-A
Country	US
Kind code	A1
Filing date	Mar 16, 2018
Priority date	Mar 17, 2017
Publication date	Apr 29, 2021
Grant date	—

Abstract

Systems and methods describe augmented reality provided for navigated surgery. An augmented reality overlay (e.g. computer generated images) is rendered and displayed over images of a tracked anatomical structure. An optical sensor unit provides tracking images of targets associated with objects including the anatomical structure in a real 3D space as well as visible images thereof. The anatomical structure is registered, generating corresponding poses of the anatomical structure in a computational 3D space from poses in the real 3D space. The overlay pose in the computational 3D space is aligned with the anatomical structure pose so that the overlay is rendered on a display of the anatomical structure in a desired pose. The overlay may be generated from a (3D) overlay model such of a generic or patient specific bone, or other anatomical structure or object. The overlay may be used to register the anatomical structure.

First claim

What is claimed is:

1 . A computer-implemented method to provide augmented reality in relation to a patient, the method comprising:
receiving, by at least one processor, images of a real 3D space containing the patient and one or more targets associated with respective objects and/or anatomical structures of the patient in the real 3D space, the images received from a single optical sensor unit having a field of view of the real 3D space containing the patient and one or more targets; determining tracking information from the images for respective ones of the one or more targets; registering an anatomical structure of the patient in a computational 3D space maintained by the at least one processor using tracking information for a respective target associated with the anatomical structure, generating a corresponding position and orientation of the anatomical structure in the computational 3D space from a position and orientation of the anatomical structure in the real 3D space; aligning an overlay model of an augmented reality overlay to a desired position and orientation in the computational 3D space relative to the corresponding position and orientation of the anatomical structure; and rendering and providing the augmented reality overlay for display on a display screen in the desired position and orientation.

2 . The method of claim 1 comprising providing the images of the real 3D space for display on the display screen to simultaneously visualize the anatomical structure and the augmented reality overlay.

3 . The method of claim 1 , wherein the optical sensor unit comprises calibration data to determine 3D measurements from the images of the real 3D space provided by the optical sensor unit in 2D and the step of determining tracking information comprises using by the at least one processor the calibration data to determine the tracking information.

4 . The method of claim 1 , comprising, in real time and in response to a relative movement of the anatomical structure and the optical sensor unit in the real 3D space, wherein a pose of the respective target associated with the anatomical structure continuously indicates a position and orientation of the anatomical structure in the real 3D space:
determining a moved position and orientation of the anatomical structure in the real 3D space using the images received from the optical sensor unit; updating the aligning of the augmented reality overlay relative to the moved position and orientation of the anatomical structure to determine a moved desired position and orientation of the augmented reality overlay; and providing the augmented reality overlay for display in the moved desired position and orientation.

5 . The method of claim 4 wherein the respective target associated with the anatomical structure is either 1) attached to the anatomical structure such that one or both of the optical sensor unit and anatomical structure are free to move in the real 3D space or 2) attached to another object while the location of anatomical structure remains constant in the real 3D space and the optical sensor unit alone is free to move in the real 3D space.

6 .- 10 . (canceled)

11 . The method of claim 1 , wherein the overlay model is a 3D model of a mechanical axis model and the augmented reality overlay is an image of a mechanical axis and/or a further axis or plane, a location of which is determined relative to a location of the mechanical axis of the anatomical structure.

12 . The method of claim 11 , comprising determining the mechanical axis of the anatomical structure using tracking information obtained from target images as the anatomical structure is rotated about an end of the anatomical structure.

13 . The method of claim 12 , wherein the further axis and/or plane is a resection plane.

14 . The method of claim 13 , wherein the location of the resection plane along the mechanical axis model is adjustable in response to user input thereby to adjust the desired position and orientation of the resection plane in the augmented reality overlay.

15 . The method of claim 11 , wherein the bone is a femur.

16 . The method of claim 15 , comprising:
registering a tibia of a same leg of the patient in the computational 3D space, the tibia coupled to a tibia target of the one or more targets, the at least one processor determining a position and orientation of the tibia in the real 3D space to generate a corresponding position and orientation of the tibia in the computational 3D space from tracking information determined from images of the tibia target; aligning a second overlay model of a second augmented reality overlay to a second desired position and orientation in the computational 3D space relative to the corresponding position and orientation of the tibia; providing the second augmented reality overlay for display on the display screen in the second desired position and orientation.

17 . The method of claim 16 , wherein registering uses images of one of the targets attached to a probe where the probe identifies first representative locations on the tibia with which to define a first end of the tibia and second identifying locations about an ankle of the patient with which to define a second end and a mechanical axis of the tibia.

18 . The method of claim 16 , comprising:
tracking movement of the position and orientation of the tibia in the real 3D space; updating the corresponding position and orientation of the tibia in response to the movement of the position and orientation of the tibia in the real 3D space; updating the aligning of the second augmented reality overlay relative to the position and orientation of the tibia as moved to determine the second desired position and orientation as moved; and providing the second augmented reality overlay for display in the second desired position and orientation as moved.

19 . The method of claim 18 , comprising determining a location of each of the augmented reality overlay of the femur and the augmented reality overlay of the tibia and indicating a relative location to one another to denote at least one of proximity and intersection.

20 . (canceled)

21 . The method of claim 1 , wherein the anatomical structure is surgically modified and wherein the overlay model is a 3D model of a generic or patient-specific human anatomical structure prior to replacement by a prosthetic implant and the augmented reality overlay is an image representing a generic or a patient-specific human anatomical structure respectively; and wherein the method comprises providing images of the patient for display on the display screen to simultaneously visualize the anatomical structure and the augmented reality overlay.

22 . The method of claim 1 , wherein the overlay model is a 3D model defined from pre-operative images of the patient.

23 . The method of claim 1 , wherein the overlay model is a 3D model defined from pre-operative images of the patient and the pre-operative images of the patient show a diseased human anatomical structure and wherein the overlay model represents the diseased human anatomical structure without a disease.

24 . A computer-implemented method to provide augmented reality in relation to a patient, the method comprising:
receiving, by at least one processor, images of a real 3D space containing the patient and one or more targets associated with respective objects and/or anatomical structures of the patient in the real 3D space, the images received from a single optical sensor unit havi

Assignees

• Intellijoint Surgical Inc

Inventors

• Fanson Richard Tyler
• Hladio Andre Novomir
• Schwarzkopf Ran
• Smith Jonathan
• Becker Luke Adrian Weber

Classifications

• A61B2034/2057

  Details of tracking cameras · CPC title

• A61B2090/371

  with simultaneous use of two cameras · CPC title

• A61B34/20 Primary

  Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis · CPC title

• A61B2034/2065

  Tracking using image or pattern recognition · CPC title

• A61B2034/2048

  using an accelerometer or inertia sensor · CPC title