Method for real time update of fly-through camera placement

Patent metadata

Field	Value
Publication number	US-11656735-B2
Application number	US-202117528429-A
Country	US
Kind code	B2
Filing date	Nov 17, 2021
Priority date	Oct 22, 2018
Publication date	May 23, 2023
Grant date	May 23, 2023

Abstract

A virtual endoscopic view shows a surgical area and surrounding anatomy and may also show a position of a surgical instrument in use during a surgical procedure, allowing a surgeon to virtually view the surgical area when direct viewing or actual endoscopic views are incomplete, obstructed, or otherwise unavailable or undesirable. In order to render the endoscopic view, an IGS navigation system may be configured with an observer point and an observer orientation within 3-D space based upon user inputs. A user interface for defining these points allows a user to view a virtual endoscopic preview in real-time while providing inputs, thus improving the likelihood that the resulting virtual endoscopic view is as desired by the user; and reducing time spent redefining and reconfiguring the virtual endoscopic view. The virtual endoscopic preview may provide combinations of static and dynamic images to illustrate the spatial relationship of the provided inputs.

First claim

We claim:

1. A system comprising:
a) a display;
b) user input;
c) a set of preoperative images associated with a patient; and
d) a processor configured to provide a virtual camera placement interface to a user via the display and receive inputs via the user input; wherein the virtual camera placement interface comprises a set of preoperative image panes and a virtual camera view, wherein each of the set of preoperative image panes corresponds to a distinct viewpoint and comprises a preoperative image selected from the set of preoperative images based on the distinct viewpoint, and wherein the user input is operable to move a cursor over and make selections from the set of preoperative image panes,
e) wherein the processor is further configured to:
i) define a first point based upon a first selection received via the user input, wherein the first selection comprises a point on one of the set of preoperative image panes that corresponds to a first cursor position of the cursor when the first selection is received and the distinct viewpoint corresponding to the preoperative image pane on which the first cursor position is located;
ii) define a second point based upon a second cursor position of the cursor located on any of the set of preoperative image panes and the distinct viewpoint corresponding to the preoperative image pane on which the second cursor position is located;
iii) display a real-time virtual endoscopic preview in the virtual camera view based upon the first point and the second point, wherein the real-time virtual endoscopic preview is a virtual camera view having a location defined by a point from a group consisting of the first point and the second point, and an orientation defined by a different point selected from the group consisting of the first point and the second point;
iv) after displaying the real-time virtual endoscopic preview, receive a second selection via the user input, wherein the second selection comprises a selected second point determined based upon the second point;
v) based upon a modify selection received via the user input, discard one of the first point or the selected second point;
vi) define a modified point based upon a third cursor position, wherein the modified point is the point that was discarded; and
vii) display the real-time virtual endoscopic preview in the virtual camera view based upon a retained point and the modified point, wherein the retained point is the point that was not discarded.

2. The system of claim 1 , wherein the processor is further configured to:
a) change the second point as the cursor moves and the second cursor position changes, and
b) update the real-time virtual endoscopic preview as the second point is changed.

3. The system of claim 1 , wherein the processor is further configured to:
a) after displaying the real-time virtual endoscopic preview, receive a second selection via the user input, wherein the second selection comprises a selected second point determined based upon the second point, and
b) create a virtual camera definition based upon the first point and the selected second point, wherein the virtual camera definition is configured to be usable by an image guided surgery navigation system to produce a virtual endoscopic view during a surgical procedure.

4. The system of claim 1 , wherein the processor is further configured to, when displaying the real-time virtual endoscopic preview:
a) determine a spatial relationship between the first point and the second point,
b) create a route through the set of preoperative images based upon the spatial relationship, and
c) display a sequence of flythrough images in the virtual camera view based upon the route.

5. The system of claim 4 , wherein the sequence of flythrough images comprises a plurality of sequential images selected from the set of preoperative images and arranged in an order that they would be viewed while traversing the route.

6. The system of claim 4 , wherein the virtual camera placement interface comprises a set of navigation controls, and wherein the processor is further configured to, perform an action in response to inputs via the set of navigation controls, wherein the action is selected from at least one of:
a) adjust a speed and order at which the sequence of flythrough images is displayed,
b) pause display of the sequence of flythrough images,
c) display a new image from the set of preoperative images in the virtual camera view based upon inputs indicating a change to one or both of the view position and orientation from an initial view position and orientation provided by the route, and
d) change one or both of the first point and the second point based upon the new image.

7. The system of claim 6 , wherein the processor is further configured to, when the new image is displayed, update one or more of the preoperative image panes to comprise a new preoperative image from the set of preoperative images, wherein the new preoperative image for each is determined based upon its proximity and relationship to the new image.

8. The system of claim 1 , wherein the processor is further configured to:
a) after displaying the real-time virtual endoscopic preview, receive a second selection via the user input, wherein the second selection comprises a selected second point determined based upon the second point,
b) based upon a swap selection received via the user input, swap the first point and the selected second point, and
c) display the real-time virtual endoscopic preview in the virtual camera view based upon the swap of the first point and the selected second point.

9. The system of claim 1 , wherein the processor is further configured to, prior to the first selection being received via the user input:
a) define the first point based upon the first cursor position located on one of the set of preoperative image panes,
b) define the second point based upon the viewpoint corresponding to the preoperative image pane on which the first cursor position is located, and
c) display the real-time virtual endoscopic preview in the virtual camera view based upon the first point and the second point.

10. A system comprising:
a) a display;
b) user input;
c) a set of preoperative images associated with a patient; and
d) a processor configured to:
i) generate a three-dimensional model of an anatomy of a patient based on the set of preoperative images; and
ii) provide a virtual camera placement interface to a user via the display and receive inputs via the user input; wherein the virtual camera placement interface comprises a set of image panes and a virtual camera view, wherein each of the set of image panes corresponds to a distinct viewpoint and comprises a rendered view of the three-dimensional model based on the distinct viewpoint, and wherein the user input is operable to move a cursor over and make selections from the set of image panes,
e) wherein the processor is further configured to:
i) define a first point based upon a first selection received via the user input, wherein the first selection comprises a point on one of the set of image panes that corresponds to a first cursor position of the cursor when the first selection is received and the distinct viewpoint corresponding to the image pane on which the first cursor position is located;
ii) define a second point based upon a second cursor position of the cursor located on any of the set of image panes and the distinct viewpoint corresponding to the image pane on which the second cursor position is located;
iii) display a real-time virtual endoscopic preview in the virtual camera view based upon the first point and the second point, wherein the real-time virtual endoscopic preview

Assignees

• Acclarent Inc
• Biosense Webster Israel Ltd

Inventors

• Palushi Jetmir
• Salazar Henry F
• Trott Jordan R
• Levi Moran
• Bustan Itamar
• Pinsky Yoav
• Racheli Noam
• Papadakis Athanasios

Classifications

• A61B17/24

  for use in the oral cavity, larynx, bronchial passages or nose (for medical inspection of cavities or tubes in the body A61B1/00); Tongue scrapers · CPC title

• G06T2200/24

  involving graphical user interfaces [GUIs] · CPC title

• G06F3/04815 Primary

  Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object · CPC title

• G16H50/50

  for simulation or modelling of medical disorders · CPC title

• A61B2034/105

  Modelling of the patient, e.g. for ligaments or bones · CPC title