Method for real time update of fly-through camera placement

Patent metadata

Field	Value
Publication number	US-11204677-B2
Application number	US-201916577019-A
Country	US
Kind code	B2
Filing date	Sep 20, 2019
Priority date	Oct 22, 2018
Publication date	Dec 21, 2021
Grant date	Dec 21, 2021

Abstract

A virtual endoscopic view shows a surgical area and surrounding anatomy and may also show a position of a surgical instrument in use during a surgical procedure, allowing a surgeon to virtually view the surgical area when direct viewing or actual endoscopic views are incomplete, obstructed, or otherwise unavailable or undesirable. In order to render the endoscopic view, an IGS navigation system may be configured with an observer point and an observer orientation within 3-D space based upon user inputs. A user interface for defining these points allows a user to view a virtual endoscopic preview in real-time while providing inputs, thus improving the likelihood that the resulting virtual endoscopic view is as desired by the user; and reducing time spent redefining and reconfiguring the virtual endoscopic view. The virtual endoscopic preview may provide combinations of static and dynamic images to illustrate the spatial relationship of the provided inputs.

First claim

We claim:

1. A system comprising:
(a) a display;
(b) user input;
(c) a set of preoperative images associated with a patient; and
(d) a processor configured to provide a virtual camera placement interface to a user via the display and receive inputs via the user input;
wherein the virtual camera placement interface comprises a set of preoperative image panes and a virtual camera view, wherein each of the set of preoperative image panes corresponds to a viewpoint, wherein the set of preoperative image panes comprises preoperative image panes corresponding to a top-down viewpoint, a side viewpoint, and a frontal viewpoint, wherein each of the set of preoperative image panes comprises a preoperative image from the set of preoperative images displayed from the viewpoint corresponding to the preoperative image pane which comprises it, and wherein the user input is operable to move a cursor over and make selections from the set of preoperative image panes,
wherein the processor is further configured to:
(i) define a first point based upon a first selection received via the user input, wherein the first selection comprises a point on one of the set of preoperative image panes that corresponds to a first cursor position of the cursor when the first selection is received and is based upon the viewpoint corresponding to the preoperative image pane comprising the point corresponding to the first cursor position of the cursor when the first selection is received,
(ii) define a second point based upon (A) a second cursor position of the cursor located on any of the set of preoperative image panes, and (B) the viewpoint corresponding to the preoperative image pane on which the second cursor position is located, and
(iii) display a real-time virtual endoscopic preview in the virtual camera view based upon the first point and the second point.

2. The system of claim 1 , wherein the processor is further configured to:
(i) change the value of the second point as the cursor moves and the second cursor position changes, and
(ii) update the real-time virtual endoscopic preview as the second point is changed.

3. The system of claim 1 , wherein the processor is further configured to:
(i) after displaying the real-time virtual endoscopic preview, receive a second selection via the user input, wherein the second selection comprises a selected second point determined based upon the second point, and
(ii) create a virtual camera definition based upon the first point and the selected second point, wherein the virtual camera definition is configured to be usable by an image guided surgery navigation system to produce a virtual endoscopic view during a surgical procedure.

4. The system of claim 3 , wherein the system comprises the image guided surgery navigation system, and wherein the user input comprises a pointing device.

5. The system of claim 1 , wherein the processor is further configured to, when displaying the real-time virtual endoscopic preview:
(i) determine a spatial relationship between the first point and the second point,
(ii) create a route through the set of preoperative images based upon the spatial relationship, and
(iii) display a sequence of flythrough images in the virtual camera view based upon the route.

6. The system of claim 5 , wherein the sequence of flythrough images comprises a plurality of sequential images selected from the set of preoperative images and arranged in the order that they would be viewed while traversing the route.

7. The system of claim 5 , wherein the virtual camera placement interface comprises a set of navigation controls, and wherein the processor is further configured to, in response to inputs via the set of navigation controls, adjust the speed and order at which the sequence of flythrough images is displayed.

8. The system of claim 7 , wherein the processor is further configured to, in response to inputs via the set of navigations controls:
(i) pause display of the sequence of flythrough images,
(ii) display a new image from the set of preoperative images in the virtual camera view based upon inputs indicating a change to one or both of the view position and orientation from an initial view position and orientation provided by the route, and
(iii) change the value of one or both of the first point and the second point based upon the new image.

9. The system of claim 8 , wherein the processor is further configured to, when the new image is displayed, update one or more of the preoperative image panes to comprise a new preoperative image from the set of preoperative images, wherein the new preoperative image for each is determined based upon its proximity and relationship to the new image.

10. The system of claim 5 , wherein the first point is associated with a location of a virtual camera and the second point is associated with an orientation of the virtual camera, and wherein the route comprises a start point that is determined based upon the first point and an end point that is determined based upon the second point.

11. The system of claim 5 , wherein the second point is associated with a location of a virtual camera and the first point is associated with an orientation of the virtual camera, and wherein the route comprises a start point that is determined based upon the second point and an end point that is determined based upon the first point.

12. The system of claim 1 , wherein the processor is further configured to:
(i) after displaying the real-time virtual endoscopic preview, receive a second selection via the user input, wherein the second selection comprises a selected second point determined based upon the second point,
(ii) display the real-time virtual endoscopic preview in the virtual camera view based on a virtual camera located at the first point and oriented toward the selected second point,
(iii) based upon a swap selection received via the user input, swap the values of the first point and the selected second point, and
(iv) display the real-time virtual endoscopic preview in the virtual camera view based upon a virtual camera located at the changed first point and oriented toward the changed selected second point.

13. The system of claim 1 , wherein the processor is further configured to:
(i) after displaying the real-time virtual endoscopic preview, receive a second selection via the user input, wherein the second selection comprises a selected second point determined based upon the second point,
(ii) based upon a modify selection received via the user input, discard the selected value of one of the first point or the selected second point,
(iii) define a modified point based upon a third cursor position, wherein the modified point is the point whose value was discarded, and
(iv) display the real-time virtual endoscopic preview in the virtual camera view based upon a retained point and the modified point, wherein the retained point is the point whose value was not discarded.

14. The system of claim 1 , wherein the processor is further configured to, prior to the first selection being received via the user input:
(i) define the first point based upon the first cursor position located on one of the set of preoperative image panes,
(ii) define the second point based upon the viewpoint corresponding to the preoperative image pane on which the first cursor position is located, and
(iii) display the real-time virtual endoscopic preview in the virtual camera view based upon the first point and the second point.

15. A method for configuring a virtual camera in 3-D space with a virtual camera placement interface, comprising the steps:
(a) disp

Assignees

• Acclarent Inc
• Biosense Webster Israel Ltd

Inventors

• Palushi Jetmir
• Salazar Henry F
• Trott Jordan R
• Levi Moran
• Bustan Itamar
• Pinsky Yoav
• Racheli Noam
• Papadakis Athanasios

Classifications

• A61B2090/365

  augmented reality, i.e. correlating a live optical image with another image · CPC title

• G06F3/04815 Primary

  Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object · CPC title

• A61B90/37

  Surgical systems with images on a monitor during operation · CPC title

• G06F2203/04803

  Split screen, i.e. subdividing the display area or the window area into separate subareas · CPC title

• A61B2090/3762

  using computed tomography systems [CT] · CPC title