Systems, methods, and apparatus for diagnostic inferencing with a multimodal deep memory network

Patent metadata

Field	Value
Publication number	US-11621075-B2
Application number	US-201716330174-A
Country	US
Kind code	B2
Filing date	Sep 5, 2017
Priority date	Sep 7, 2016
Publication date	Apr 4, 2023
Grant date	Apr 4, 2023

Abstract

The described embodiments relate to systems, methods, and apparatus for providing a multimodal deep memory network ( 200 ) capable of generating patient diagnoses ( 222 ). The multimodal deep memory network can employ different neural networks, such as a recurrent neural network and a convolution neural network, for creating embeddings ( 204, 214, 216 ) from medical images ( 212 ) and electronic health records ( 206 ). Connections between the input embeddings ( 204 ) and diagnoses embeddings ( 222 ) can be based on an amount of attention that was given to the images and electronic health records when creating a particular diagnosis. For instance, the amount of attention can be characterized by data ( 110 ) that is generated based on sensors that monitor eye movements of clinicians observing the medical images and electronic health records. Resulting patient diagnoses can be provided according to a predetermined classification of weights, or a compilation of words that are generated over multiple iterations of the multimodal deep memory network.

First claim

We claim:

1. A method of querying a trained memory network performed by one or more processors, the method comprising:
generating a first set of embeddings from an image received at a first neural network;
generating a second set of embeddings from a document received at a second neural network, wherein each of the image and the document are associated with a medical patient;
applying the first set of embeddings and the second set of embeddings as input across the trained memory network, the memory network including a key-value memory and including multiple different diagnosis embeddings, the key-value memory including memory-slots each according to a key embedding and a value embedding;
generating weights for the multiple different diagnosis embeddings based on a correlation between the first set of embeddings and the second set of embeddings, and the key embeddings and the value embeddings; and
providing a patient diagnosis for the medical patient at least based on the generated weights for the multiple different diagnosis embeddings.

2. The method of claim 1 , wherein applying the first set of embeddings and the second set of embeddings as input across the trained memory network comprises iteratively updating an input embedding based on key embeddings and value embeddings of memory-slots of the key-value memory to obtain the correlation between the first set of embeddings and the second set of embeddings, and the key embeddings and the value embeddings.

3. The method of claim 2 , wherein updating the input embedding based on key embeddings and value embeddings of memory-slots of the key-value memory comprises determining a weight of a value embedding of a memory-slot, said weight measuring a similarity of a key embedding of the memory-slot to the input embedding, and computing an updated input embedding based on the weight and the value embedding.

4. The method of claim 3 , wherein determining the weight of a value embedding of a memory-slot comprises calculating the weight using a multi-layer feedforward neural network.

5. The method of claim 2 , wherein generating a weight for a diagnosis embedding comprises applying a sigmoid function to the iteratively updated input embedding and the diagnosis embedding.

6. The method of claim 1 , wherein the memory network includes weights that are based at least in part on an amount of attention given to a portion of medical data from which the key embeddings and the value embeddings were generated.

7. The method of claim 1 , wherein the amount of attention given to a portion of medical data comprises data indicating an electronic health record that a clinician accessed when making a diagnosis.

8. The method of claim 1 , wherein the amount of attention corresponds to attention data that is based on an amount of eye movement exhibited by a user accessing the medical data.

9. The method of claim 1 , wherein generating the second set of embeddings includes generating an input value embedding from a section heading of the document, and generating input key embeddings from content that is separate from the section heading of the document.

10. The method of claim 1 , wherein key embeddings and data embeddings of the key-value memory are generated at least in part based on medical data from a medical-related website or a database for assisting clinicians with determining patient diagnoses.

11. The method of claim 1 , wherein key embeddings and data embeddings of the key-value network are generated at least in part from a medical image with a corresponding textual description.

12. The method of claim 1 , wherein the second neural network is a bi-directional recurrent neural network, and the document corresponds to an electronic health record.

13. The method of claim 1 , wherein the first neural network is a convolutional neural network.

14. A computing device for querying a trained memory network, comprising:
one or more processors; and
memory configured to store instructions that, when executed by the one or more processors cause the one or more processors to perform steps that include:
generating a first set of embeddings from an image received at a first neural network;
generating a second set of embeddings from a document received at a second neural network, wherein each of the image and the document are associated with a medical patient;
applying the first set of embeddings and the second set of embeddings as input across the trained memory network, the memory network including a key-value memory and including multiple different diagnosis embeddings, the key-value memory including memory-slots each according to a key embedding and a value embedding;
generating weights for the multiple different diagnosis embeddings based on a correlation between the first set of embeddings and the second set of embeddings, and the key embeddings and the value embeddings; and
providing a patient diagnosis for the medical patient at least based on the generated weights for the multiple different diagnosis embeddings.

15. A non-transitory computer-readable medium configured to store instructions for querying a trained memory network that, when executed by one or more processors, cause the one or more processors to perform steps that include:
generating a first set of embeddings from an image received at a first neural network;
generating a second set of embeddings from a document received at a second neural network, wherein each of the image and the document are associated with a medical patient;
applying the first set of embeddings and the second set of embeddings as input across the trained memory network, the memory network including a key-value memory and including multiple different diagnosis embeddings, the key-value memory including memory-slots each according to a key embedding and a value embedding;
generating weights for the multiple different diagnosis embeddings based on a correlation between the first set of embeddings and the second set of embeddings, and the key embeddings and the value embeddings; and
providing a patient diagnosis for the medical patient at least based on the generated weights for the multiple different diagnosis embeddings.

16. The non-transitory computer-readable medium of claim 1 , wherein the amount of attention is based attention data that is based on an amount of eye movement exhibited by a user accessing the medical data.

17. The non-transitory computer-readable medium of claim 1 , wherein generating the second set of embeddings includes generating an input value embedding from a section heading of the document, and generating input key embeddings from content that is separate from the section heading of the document.

18. The non-transitory computer-readable medium of claim 1 , wherein the medical data, from which the key embeddings and the value embeddings are generated, includes a medical image with a corresponding textual description.

19. The method of claim 1 , further including: measuring a probability that a document, an image, a portion of the document, or a portion of the image is similar to each existing key.

20. The method of claim 19 , further including: attaching each of the probabilities to a corresponding one of the memory slots.

Assignees

• Koninklijke Philips Nv

Inventors

• Al Hasan Sheikh Sadid
• Zhao Siyuan
• Farri Oladimeji Feyisetan
• Lee Kathy Mi Young
• Datla Vivek
• Qadir Ashequl
• Liu Junyi
• Prakash Aaditya

Classifications

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/0442

  characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G16H50/20 Primary

  for computer-aided diagnosis, e.g. based on medical expert systems · CPC title