Systems and methods for visualizing anatomical structure of patient during surgery

Patent metadata

Field	Value
Publication number	US-2024074811-A1
Application number	US-202217930058-A
Country	US
Kind code	A1
Filing date	Sep 6, 2022
Priority date	Sep 6, 2022
Publication date	Mar 7, 2024
Grant date	—

Abstract

A system and method for visualizing anatomical structure of a patient during a surgery are provided. An anatomical image and a first optical image of a patient may be obtained. The anatomical image may be captured by performing a medical scan on the patient before a surgery of the patient, and the first optical image may be captured during the surgery. A target image may be generated by combining the anatomical image and the first optical image. The target image may be rendered based on a viewpoint of a target operator of the surgery. A display device may be directed to display the rendered target image.

First claim

What is claimed is:

1 . A system, comprising:
at least one storage device including a set of instructions; and at least one processor configured to communicate with the at least one storage device, wherein when executing the set of instructions, the at least one processor is configured to direct the system to perform operations including:
obtaining an anatomical image and a first optical image of a patient, the anatomical image being captured by performing a medical scan on the patient before a surgery of the patient, and the first optical image being captured during the surgery;
generating a target image by combining the anatomical image and the first optical image;
rendering the target image based on a viewpoint of a target operator of the surgery; and
directing a display device to display the rendered target image.

2 . The system of claim 1 , wherein the generating a target image by combining the anatomical image and the first optical image includes:
obtaining a reference optical image of the patient captured during the medical scan; transforming the anatomical image based on the reference optical image and the first optical image such that a posture of the patient in the transformed anatomical image aligns with the posture of the patient in the first optical image; and generating the target image by combining the transformed anatomical image and the first optical image.

3 . The system of claim 2 , wherein the transforming the anatomical image based on the reference optical image and the first optical image includes:
determining a first transformation relationship between the reference optical image and the anatomical image; determining a second transformation relationship between the reference optical image and the first optical image; and transforming the anatomical image based on the first transformation relationship and the second transformation relationship such that the posture of the patient in the transformed anatomical image aligns with the posture of the patient in the first optical image.

4 . The system of claim 3 , wherein the determining a second transformation relationship between the reference optical image and the first optical image includes:
determining, based on the reference optical image, a reference representation relating to the posture of the patient in the reference optical image; determining, based on the first optical image, a target representation relating to the posture of the patient in the first optical image; and determining, based on the reference representation and the target representation of the patient, the second transformation relationship.

5 . The system of claim 4 , wherein the reference representation and the target representation of the patient are obtained using a representation determination model.

6 . The system of claim 2 , wherein the generating the target image by combining the transformed anatomical image and the first optical image includes:
determining, based on the viewpoint of the target operator, a field of view (FOV) of the target operator; determining a target portion of the transformed anatomical image corresponding to at least a portion of the patient within the FOV of the target operation; and generating the target image by overlaying the target portion of the transformed anatomical image on the first optical image.

7 . The system of claim 1 , wherein determining the viewpoint of the target operator includes:
determining facial information of the target operator based on a second optical image of the target operator; and determining, based on the facial information, the viewpoint of the target operator.

8 . The system of claim 1 , wherein the rendering the target image based on a viewpoint of a target operator of the surgery includes:
obtaining a third optical image of a surgical device used in the surgery captured when the first optical image is captured; processing the target image by adding a visual element representing the surgical device on the target image based on the third optical image of the surgical device; and rendering the processed target image based on the viewpoint of the target operator.

9 . The system of claim 1 , wherein the rendering the target image based on a viewpoint of a target operator of the surgery includes:
obtaining a fourth optical image of one or more markers placed on the patient captured when the first optical image is captured; and processing the target image by adding a visual element representing the one or more markers on the target image based on the fourth optical image of the one or more markers; and rendering the processed target image based on the viewpoint of the target operator.

10 . The system of claim 1 , wherein the viewpoint of the target operator is set by the target operator.

11 . A method, the method being implemented on a computing device having at least one storage device and at least one processor, the method comprising:
obtaining an anatomical image and a first optical image of a patient, the anatomical image being captured by performing a medical scan on the patient before a surgery of the patient, and the first optical image being captured during the surgery; generating a target image by combining the anatomical image and the first optical image; rendering the target image based on a viewpoint of a target operator of the surgery; and directing a display device to display the rendered target image.

12 . The method of claim 11 , wherein the generating a target image by combining the anatomical image and the first optical image includes:
obtaining a reference optical image of the patient captured during the medical scan; transforming the anatomical image based on the reference optical image and the first optical image such that a posture of the patient in the transformed anatomical image aligns with the posture of the patient in the first optical image; and generating the target image by combining the transformed anatomical image and the first optical image.

13 . The method of claim 12 , wherein the transforming the anatomical image based on the reference optical image and the first optical image includes:
determining a first transformation relationship between the reference optical image and the anatomical image; determining a second transformation relationship between the reference optical image and the first optical image; and transforming the anatomical image based on the first transformation relationship and the second transformation relationship such that the posture of the patient in the transformed anatomical image aligns with the posture of the patient in the first optical image.

14 . The method of claim 13 , wherein the determining a second transformation relationship between the reference optical image and the first optical image includes:
determining, based on the reference optical image, a reference representation relating to the posture of the patient in the reference optical image; determining, based on the first optical image, a target representation relating to the posture of the patient in the first optical image; and determining, based on the reference representation and the target representation of the patient, the second transformation relationship.

15 . The method of claim 14 , wherein the reference representation and the target representation of the patient are obtained using a representation determination model.

16 . The method of claim 12 , wherein the generating the target image by combining the transformed anatomical image and the first optical image includes:
de

Assignees

• Shanghai United Imaging Intelligence Co Ltd

Inventors

• Wu Ziyan
• Planche Benjamin
• Zheng Meng
• Innanje Arun
• Chen Terrence

Classifications

• G06T2207/20212

  Image combination · CPC title

• G06T2207/30004

  Biomedical image processing · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T2207/10116

  X-ray image · CPC title

• G06T2207/10088

  Magnetic resonance imaging [MRI] · CPC title