Interpretation of intraoperative sensor data using concept graph neural networks

Patent metadata

Field	Value
Publication number	US-12362068-B2
Application number	US-202218550367-A
Country	US
Kind code	B2
Filing date	Mar 28, 2022
Priority date	Mar 26, 2021
Publication date	Jul 15, 2025
Grant date	Jul 15, 2025

Abstract

Systems and methods are provided for generating a statistical parameter representing a state of a surgical procedure from sensor data. Sensor data representing a time period. is received from a sensor. Numerical features representing the time period are generated from the sensor data. Each of a plurality of long short term memory units are updated according to the plurality of numerical features via a message passing process. The long short term memory units are connected to form a graph, with a first set of the long short term memory units representing a plurality of nodes of the graph and a second set of the long short term memory units representing a plurality of hyperedges of the graph. A statistical parameter representing a state of the surgical procedure for the time period is derived from an output of one of the long short term memory units and provided to a user.

First claim

What is claimed is:

1. A system comprising:
a sensor positioned to monitor a surgical procedure on a patient such that the sensor senses objects within a scene that are associated with specific surgical states or world states to generate sensor data;
a processor; and
a non-transitory computer readable medium stores machine executable instructions for providing a statistical parameter representing a state of the surgical procedure for the time period, the machine executable instructions being executed by the processor to provide:
a sensor interface that receives the sensor data from the sensor, the sensor data representing a time period of a plurality of time periods comprising the surgical procedure;
a feature extractor that generates a plurality of numerical features representing the time period from the sensor data; and
a plurality of temporal processing network modules, each configured to receive the extracted plurality of numerical features, and the plurality of temporal processing network modules being connected to form a graph, with a first set of the plurality of temporal processing network modules representing a plurality of nodes of the graph, a second set of the plurality of temporal processing network modules representing a plurality of hyperedges of the graph, and each of the plurality of temporal processing network modules being updated via a message passing process according to the location of the temporal processing network module within the graph, wherein the graph is a directed graph, such that the message passing process differentiates between an input node and an output node connected by at least one hyperedge of the plurality of hyperedges during updating of the plurality of temporal processing network modules; and

an output device that provides a statistical parameter representing a state of the surgical procedure for the time period to a user, the statistical parameter representing the state of the surgical procedure for the time period being derived from an output of one of the plurality of temporal processing network modules.

2. The system of claim 1 , wherein the at least one sensor comprises a camera that captures frame of video.

3. The system of claim 1 , wherein the feature extractor comprises a convolutional neural network.

4. The system of claim 1 , further comprising a network interface that provides the output representing the state of the surgical procedure for the time period to a surgical assisted decision making system.

5. The system of claim 1 , wherein the statistical parameter representing the state of the surgical procedure for the time period represents a likelihood that a critical view of safety has been achieved during the surgical procedure.

6. The system of claim 1 , wherein the surgical procedure is a laparoscopic cholecystectomy, and the statistical parameter representing the state of the surgical procedure for the time period represents a value for the Parkland Grading Scale.

7. The system of claim 1 , wherein the state of each of the first set of temporal processing network modules represents a concept associated with the surgery.

8. The system of claim 7 , wherein the surgical procedure is a laparoscopic cholecystectomy, and a state of a first temporal processing network module of the first set of temporal processing network modules represents a likelihood of exposure of the cystic plate and a state of a second temporal processing network module of the first set of temporal processing network modules represents a likelihood of exposure of the cystic duct.

9. A method comprising:
receiving sensor data from a sensor positioned to monitor a surgical procedure on a patient, wherein the sensor senses objects within a scene that are associated with specific surgical states or world states to generate the sensor data, the sensor data representing a time period of a plurality of time periods comprising the surgical procedure;
generating a plurality of numerical features representing the time period from the sensor data;
updating each of a plurality of temporal processing network modules according to the plurality of numerical features, the plurality of temporal processing network modules being connected to form a graph, with a first set of the plurality of temporal processing network modules representing a plurality of nodes of the graph, a second set of the plurality of temporal processing network modules representing a plurality of hyperedges of the graph, and each of the plurality of temporal processing network modules being updated via a message passing process according to the location of the temporal processing network module within the graph, wherein the graph is a directed graph, such that the message passing process differentiates between an input node and an output node connected by at least one hyperedge of the plurality of hyperedges during updating of the plurality of temporal processing network modules; and
providing a statistical parameter representing a state of the surgical procedure for the time period to a user at an output device, the statistical parameter representing the state of the surgical procedure for the time period being derived from an output of one of the plurality of temporal processing network modules.

10. The method of claim 9 , wherein the surgical procedure is a laparoscopic cholecystectomy, and the statistical parameter representing the state of the surgical procedure for the time period represents a value for the Parkland Grading Scale.

11. The method of claim 9 , further comprising provides the output representing the state of the surgical procedure for the time period to a surgical assisted decision making system via a network interface.

12. The method of claim 9 , wherein the output of each of the first set of temporal processing network modules represents a concept associated with the surgery, such that the output of each of the first set of temporal processing network modules is interpretable by a human user to provide information about the surgical procedure.

13. The method of claim 9 , wherein the statistical parameter representing the state of the surgical procedure for the time period represents a likelihood that a critical view of safety has been achieved during the surgical procedure.

14. The method of claim 9 , wherein receiving sensor data from the sensor positioned to monitor the surgical procedure on a patient comprises receiving frames of video from a camera.

15. A method comprising:
receiving a frame of video from a sensor positioned to monitor a surgical procedure on a patient, wherein the sensor senses objects within a scene that are associated with specific surgical states or world states to generate the frame of video, the frame of video representing a time period of a plurality of time periods comprising the surgical procedure;
generating a plurality of numerical features representing the frame of video at a convolutional neural network;
updating each of a plurality of long short term memory units according to the plurality of numerical features, the plurality of long short term memory units being connected to form a graph, with a first set of the plurality of long short term memory units representing a plurality of nodes of the graph, a second set of the plurality of long short term memory units representing a plurality of hyperedges of the graph, and each of the plurality of long short term memory units being updated via a message passing process according to the location of the long short term memory unit within the graph, wherein the graph is a directed graph, such that the message passing process differentiates be

Assignees

• Massachusetts Gen Hospital
• Massachusetts Inst Technology

Inventors

• Meireles Ozanan R
• Ban Yutong
• Hashimoto Daniel A
• Rosman Guy
• Ward Thomas
• Rus Daniela

Classifications

• G06N3/0442

  characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/09

  Supervised learning · CPC title

• G16H40/63

  for local operation · CPC title

• G16H30/40

  for processing medical images, e.g. editing · CPC title