Augmented reality interface for assisting a user to operate an ultrasound device

Patent metadata

Field	Value
Publication number	US-11861887-B2
Application number	US-202117468633-A
Country	US
Kind code	B2
Filing date	Sep 7, 2021
Priority date	Jun 20, 2016
Publication date	Jan 2, 2024
Grant date	Jan 2, 2024

Abstract

Aspects of the technology described herein relate to techniques for guiding an operator to use an ultrasound device. Thereby, operators with little or no experience operating ultrasound devices may capture medically relevant ultrasound images and/or interpret the contents of the obtained ultrasound images. For example, some of the techniques disclosed herein may be used to identify a particular anatomical view of a subject to image with an ultrasound device, guide an operator of the ultrasound device to capture an ultrasound image of the subject that contains the particular anatomical view, and/or analyze the captured ultrasound image to identify medical information about the subject.

First claim

What is claimed is:

1. An apparatus, comprising:
at least one processor configured to:
obtain an ultrasound image of a subject;
identify at least one anatomical feature of the subject in the ultrasound image using an automated image processing technique;
identify a value of an ejection fraction of the subject using the at least one anatomical feature in the ultrasound image; and
form a composite image including the ultrasound image and the value of the ejection fraction,

wherein the at least one processor is further configured to obtain the ultrasound image by guiding an operator of an ultrasound device to capture the ultrasound image of the subject,
wherein guiding the operator of the ultrasound device comprises providing the ultrasound image as an input to a first multi-layer neural network, and
wherein the at least one processor is further configured to identify the at least one anatomical feature of the subject by providing the ultrasound image as an input to a second multi-layer neural network that is different from the first multi-layer neural network.

2. The apparatus of claim 1 ,
wherein the at least one anatomical feature is a heart ventricle, and
wherein the at least one processor is configured to identify the heart ventricle of the subject at least in part by analyzing the ultrasound image using the second multi-layer neural network comprising at least one layer selected from the group consisting of: a pooling layer, a rectified linear units (ReLU) layer, a convolution layer, a dense layer, a pad layer, a concatenate layer, and an upscale layer.

3. The apparatus of claim 1 ,
wherein the at least one processor is configured to overlay the value of the ejection fraction onto the ultrasound image of the subject to form the composite image.

4. A method, comprising:
using at least one computing device comprising at least one processor to perform:
obtaining an ultrasound image of a subject captured by an ultrasound device;
identifying at least one anatomical feature of the subject in the ultrasound image using an automated image processing technique;
identifying a value of an ejection fraction of the subject using the at least one anatomical feature in the ultrasound image; and
forming a composite image including the ultrasound image and the value of the ejection fraction,

wherein obtaining the ultrasound image comprises guiding an operator of the ultrasound device to capture the ultrasound image of the subject,
wherein guiding the operator of the ultrasound device comprises providing the ultrasound image as an input to a first multi-layer neural network, and
wherein identifying the at least one anatomical feature of the subject comprises providing the ultrasound image as an input to a second multi-layer neural network that is different from the first multi-layer neural network.

5. The method of claim 4 ,
wherein identifying the value of the ejection fraction of the subject comprises providing the ultrasound image as an input to the second multi-layer neural network.

6. The method of claim 5 ,
wherein identifying the value of the ejection fraction of the subject comprises analyzing the ultrasound image using the second multi-layer neural network comprising at least one layer selected from the group consisting of: a pooling layer, a rectified linear units (ReLU) layer, a convolution layer, a dense layer, a pad layer, a concatenate layer, and an upscale layer, and
wherein identifying the value of the ejection fraction comprises identifying a heart ventricle of the subject.

7. The method of claim 4 ,
wherein obtaining the ultrasound image of the subject comprises obtaining a plurality of ultrasound images of the subject, and
wherein identifying the value of the ejection fraction of the subject comprises identifying a ventricle in each of at least some of the plurality of ultrasound images using the second multi-layer neural network.

8. The method of claim 7 ,
wherein identifying the value of the ejection fraction comprises:
estimating a ventricle diameter of the ventricle in each of the at least some of the plurality of ultrasound images to obtain a plurality of ventricle diameters including a first ventricle diameter and a second ventricle diameter that is different from the first ventricle diameter;
using the first ventricle diameter to estimate an end-diastolic volume; and
using the second ventricle diameter to estimate an end-systolic volume.

9. The method of claim 4 ,
wherein forming the composite image comprises overlaying the value of the ejection fraction onto the ultrasound image.

10. The method of claim 4 ,
wherein forming the composite image comprises presenting the composite image.

11. A system, comprising:
an ultrasound device configured to capture an ultrasound image of a subject; and
a computing device communicatively coupled to the ultrasound device and configured to:
obtain the ultrasound image captured by the ultrasound device;
identify at least one anatomical feature of the subject in the ultrasound image using an automated image processing technique;
identify a value of an ejection fraction of the subject using the at least one anatomical feature in the ultrasound image; and
form a composite image including the ultrasound image and the value of the ejection fraction,

wherein the computing device is further configured to obtain the ultrasound image by guiding an operator of the ultrasound device to capture the ultrasound image of the subject,
wherein guiding the operator of the ultrasound device comprises providing the ultrasound image as an input to a first multi-layer neural network, and
wherein the computing device is further configured to identify the at least one anatomical feature of the subject by providing the ultrasound image as an input to a second multi-layer neural network that is different from the first multi-layer neural network.

12. The system of claim 11 ,
wherein the ultrasound device comprises a plurality of ultrasonic transducers.

13. The system of claim 12 ,
wherein the plurality of ultrasonic transducers comprises an ultrasonic transducer selected from the group consisting of: a capacitive micromachined ultrasonic transducer (CMUT), and a piezoelectric micromachined ultrasonic transducer (PMUT).

14. The system of claim 11 ,
wherein the computing device is a mobile smartphone or a tablet.

15. The system of claim 11 ,
wherein the computing device comprises a display, and wherein the computing device is configured to display the composite image.

Assignees

• Bfly Operations Inc

Inventors

• De Jonge Matthew
• Schneider Robert
• Elgena David
• Rothberg Alex
• Rothberg Jonathan M
• Sofka Michal
• Gafner Tomer
• Thiele Karl
• Neben Abraham

Classifications

• G06V10/82 Primary

  using neural networks · CPC title

• A61B8/02

  Measuring pulse or heart rate · CPC title

• A61B8/06

  Measuring blood flow · CPC title

• A61B8/065

  to determine blood output from the heart · CPC title

• A61B8/085

  for locating body or organic structures, e.g. tumours, calculi, blood vessels, nodules · CPC title