Real-time annotation of symptoms in telemedicine

Patent metadata

Field	Value
Publication number	US-2019328300-A1
Application number	US-201815964542-A
Country	US
Kind code	A1
Filing date	Apr 27, 2018
Priority date	Apr 27, 2018
Publication date	Oct 31, 2019
Grant date	—

Abstract

A teleconferencing system includes a first terminal configured to acquire an audio signal and a video signal. A teleconferencing server in communication with the first terminal and a second terminal is configured to receive the video signal and the audio signal from the first terminal, in real-time, and transmit the video signal and the audio signal to the second terminal. A symptom recognition server in communication with the first terminal and the teleconferencing server is configured to receive the video signal and the audio signal from the first terminal, asynchronously, analyze the video signal and the audio signal to detect one or more indicia of illness, generate a diagnostic alert on detecting the one or more indicia of illness, and transmit the diagnostic alert to the teleconferencing server for display on the second terminal.

First claim

1 . A teleconferencing system, comprising:
a first terminal including a camera and a microphone configured to acquire an audio signal and a high-quality: video signal and convert the acquired high-quality video signal into a low-quality video signal of a Nitrate that is less than a hit rate of the high-quality video signal; a teleconferencing server in communication with the first terminal and a second terminal and configured to receive the low-quality video signal and the audio signal from the first terminal, in real-time and transmit the low-quality video signal and the audio signal to the second terminal; and a symptom recognition server in communication with the first terminal and the teleconferencing server and configured to receive the high-quality video signal and the audio signal from the first terminal, asynchronously, analyze the high-quality video signal and the audio signal to detect one or more indicia of illness, generate a diagnostic alert on detecting the one or more indicia of illness, and transmit the diagnostic alert to the teleconferencing server for display on the second terminal.

2 . The system of claim 1 , wherein the symptom recognition server is configured to detect the indicia of illness from the high-quality video signal and the audio signal using multimodal recurrent neural networks.

3 . The system of claim 2 , wherein the symptom recognition server is configured to detect the indicia of illness from the high-quality video signal by:
detecting a face from the high-quality video signal; extracting action units from the detected face; detecting landmarks from the detected face; tracking the detected landmarks; performing semantic feature extraction using the tracked landmarks; and using the multimodal recurrent neural networks to detect the indicia of illness from the detected face, extracted action units, tracked landmarks, and extracted semantic features.

4 . The system of claim 2 , wherein the symptom recognition server is configured to detect the indicia of illness from the high-quality video signal by:
detecting a body posture from the high-quality video signal; tracking head movements from the high-quality video signal; and using the multimodal recurrent neural networks to detect the indicia of illness from the detected body posture and tracked head movements.

5 . The system of claim 2 , wherein the symptom recognition server is configured to detect the indicia of illness from the audio signal by:
detecting tone features from the audio signal; transcribing the audio signal to generate a transcription; performing natural language processing on the transcription; performing semantic analysis on the transcription; performing language structure extraction on the transcription; and using the recurrent neural networks to detect the indicia of illness from the detected tone features, the transcription, the results of the natural language processing, the results of the semantic analysis, and the results of the language structure extraction.

6 . The system of claim 1 , wherein the first terminal is configured to convert the video signal into a low-quality video signal of less bitrate by reducing a resolution of the high-quality signal, by reducing a framerate of the high-quality signal, or by compressing the high-quality signal.

7 . The system of claim 1 , wherein the symptom recognition server is part of or locally connected to the first terminal.

8 . The system of claim 1 , wherein the teleconferencing server is in communication with the first terminal and the second terminal over the Internet or another wide-area network.

9 . The system of claim 1 , wherein the second terminal is configured to display the low-quality video signal as part of a teleconference and the teleconferencing server is configured to overlay the diagnostic alert on the display of the second terminal.

10 . The system of claim 9 , wherein the teleconferencing server is configured to overlay the diagnostic alert on the display of the second terminal in the form of a textual alert.

11 . The system of claim 9 , wherein the teleconferencing server is configured to overlay the diagnostic alert on the display of the second terminal in the form of a graphic element that highlights or emphasizes a part of a face or body that the indicia of illness are based on.

12 . The system of claim 9 , wherein the teleconferencing server is configured to overlay the diagnostic alert on the display of the second terminal in the form of an annotation, highlighting, or other marking on a textual transcription of the audio signal.

13 . The system of claim 9 , wherein the teleconferencing server is configured to overlay the diagnostic alert on the display of the second terminal in the form of a picture-in-picture element that includes a replaying of a portion of the high-quality video signal that the indicia of illness are based on.

14 . A method for teleconferencing, comprising:
acquiring an audio signal and a video signal from a first terminal; transmitting the video signal and the audio signal to a teleconferencing server in communication with the first terminal and a second terminal; transmitting the video signal and the audio signal to a symptom recognition server in communication with the first terminal and the teleconferencing server; detecting indicia of illness from the video signal and the audio signal using multimodal recurrent neural networks; generating a diagnostic alert for the detected indicia illness; annotating the video signal with the diagnostic alert; and displaying the annotated video signal on the second terminal.

15 . The method of claim 14 , wherein detecting the indicia of illness from the video signal comprises:
detecting a face from the video signal; extracting action units from the detected face; detecting landmarks from the detected face; tracking the detected landmarks; performing semantic feature extraction using the tracked landmarks; and using the multimodal recurrent neural networks to detect the indicia of illness from the detected face, extracted action units, tracked landmarks, and extracted semantic features.

16 . The method of claim 14 , wherein detecting the indicia of illness from the video signal comprises:
detecting a body posture from the video signal; tracking head movements from the video signal; and using the multimodal recurrent neural networks to detect the indicia of illness from the detected body posture and tracked head movements.

17 . The method of claim 14 , wherein detecting the indicia of illness from the audio signal comprises;
detecting tone features from the audio signal; transcribing the audio signal to generate a transcription; performing natural language processing on the transcription; performing semantic analysis on the transcription; performing language structure extraction on the transcription; and using the recurrent neural networks to detect the indicia of illness from the detected tone features, the transcription, the results of the natural language processing, the results of the semantic analysis, and the results of the language structure extraction.

18 . The method of claim 14 , wherein a bit rate of the video signal is reduced prior to transmitting the video signal to the symptom recognition server.

19 . The method of claim 14 , wherein the video signal is up-sampled prior to detecting the indicia of illness from the video signal.

20 . A computer program product for detecting indicia of illness

Assignees

• IBM

Inventors

• Bozorgtabar Seyedbehzad
• Faux Noel
• Garnavi Rahil
• Sedai Suman

Classifications

• G16H50/20

  for computer-aided diagnosis, e.g. based on medical expert systems · CPC title

• G16H40/67

  for remote operation · CPC title

• G16H80/00

  ICT specially adapted for facilitating communication between medical practitioners or patients, e.g. for collaborative diagnosis, therapy or health monitoring · CPC title

• G16H30/40

  for processing medical images, e.g. editing · CPC title

• G06F40/30

  Semantic analysis · CPC title