Automated image acquisition for assisting a user to operate an ultrasound device

Patent metadata

Field	Value
Publication number	US-10959702-B2
Application number	US-201715626844-A
Country	US
Kind code	B2
Filing date	Jun 19, 2017
Priority date	Jun 20, 2016
Publication date	Mar 30, 2021
Grant date	Mar 30, 2021

Abstract

Aspects of the technology described herein relate to techniques for guiding an operator to use an ultrasound device. Thereby, operators with little or no experience operating ultrasound devices may capture medically relevant ultrasound images and/or interpret the contents of the obtained ultrasound images. For example, some of the techniques disclosed herein may be used to identify a particular anatomical view of a subject to image with an ultrasound device, guide an operator of the ultrasound device to capture an ultrasound image of the subject that contains the particular anatomical view, and/or analyze the captured ultrasound image to identify medical information about the subject.

First claim

What is claimed is:

1. An apparatus, comprising:
at least one processor configured to:
obtain an ultrasound image of a subject captured by an ultrasound device and determine, using an automated image processing technique, whether the ultrasound image contains a target anatomical view;
generate a first guidance plan in furtherance of capturing a second ultrasound image of the subject that contains the target anatomical view responsive to a determination that the ultrasound image does not contain the target anatomical view;
provide at least one instruction to an operator of the ultrasound device indicating how to reposition the ultrasound device as part of the first guidance plan;
determine that an action taken by the operator of the ultrasound device does not match the at least one instruction provided to the operator of the ultrasound device as part of the first guidance plan; and
generate a second guidance plan responsive to determining that the action taken by the operator of the ultrasound device does not match the at least one instruction provided to the operator of the ultrasound device as part of the first guidance plan.

2. The apparatus of claim 1 , wherein the at least one processor is configured to determine whether the ultrasound image contains the target anatomical view at least in part by analyzing the ultrasound image using a deep learning technique.

3. The apparatus of claim 2 , wherein the at least one processor is configured to determine whether the ultrasound image contains the target anatomical view at least in part by providing the ultrasound image as an input to a multi-layer neural network.

4. The apparatus of claim 3 , wherein the at least one processor is configured to determine whether the ultrasound image contains the target anatomical view at least in part by using the multi-layer neural network to obtain an output that is indicative of an anatomical view contained in the ultrasound image.

5. The apparatus of claim 2 , wherein the at least one processor is configured to determine whether the ultrasound image contains the target anatomical view at least in part by analyzing the ultrasound image using a multi-layer neural network comprising at least one layer selected from the group consisting of: a pooling layer, a rectified linear units (ReLU) layer, a convolution layer, a dense layer, a pad layer, a concatenate layer, and an upscale layer.

6. The apparatus of claim 1 , wherein the at least one processor is configured to determine whether the ultrasound image contains the target anatomical view at least in part by:
identifying an anatomical view contained in the ultrasound image using the automated image processing technique; and
determining whether the anatomical view contained in the ultrasound image matches the target anatomical view.

7. The apparatus of claim 6 , wherein the at least one processor is configured to, responsive to a determination that the anatomical view contained in the ultrasound image does not match the target anatomical view, generate the at least one instruction.

8. The apparatus of claim 1 , wherein the at least one processor is configured to:
provide an indication to the operator that the ultrasound device is properly positioned responsive to a determination that the ultrasound image contains the target anatomical view.

9. The apparatus of claim 8 , further comprising a display coupled to the at least one processor and configured to display the at least one instruction to the operator.

10. The apparatus of claim 8 , wherein the at least one processor is configured to provide the at least one instruction at least in part by providing an instruction to move the ultrasound device in a translational direction and/or a rotational direction.

11. The apparatus of claim 8 , wherein the at least one processor is configured to provide the at least one instruction to the operator at least in part by providing the at least one instruction to the subject.

12. A method, comprising:
using at least one computing device comprising at least one processor to perform:
obtaining an ultrasound image of a subject captured by an ultrasound device;
determining, using an automated image processing technique, whether the ultrasound image contains a target anatomical view;
responsive to determining that the ultrasound image does not contain the target anatomical view, generating a first guidance plan in furtherance of capturing a second ultrasound image of the subject that contains the target anatomical view;
providing at least one instruction to an operator of the ultrasound device indicating how to reposition the ultrasound device as part of the first guidance plan;
determining that an action taken by the operator of the ultrasound device does not match the at least one instruction provided to the operator of the ultrasound device as part of the first guidance plan;
generating a second guidance plan responsive to determining that the action taken by the operator of the ultrasound device does not match the at least one instruction provided to the operator of the ultrasound device as part of the first guidance plan; and
responsive to determining that the ultrasound image contains the target anatomical view, providing an indication to the operator that the ultrasound device is properly positioned.

13. The method of claim 12 , wherein determining whether the ultrasound image contains the target anatomical view comprises analyzing the ultrasound image using a deep learning technique.

14. The method of claim 13 , wherein determining whether the ultrasound image contains the target anatomical view comprises providing the ultrasound image as an input to a multi-layer neural network.

15. The method of claim 14 , wherein determining whether the ultrasound image contains the target anatomical view comprises using the multi-layer neural network to obtain an output that is indicative of an anatomical view contained in the ultrasound image.

16. The method of claim 13 , wherein determining whether the ultrasound image contains the target anatomical view comprises analyzing the ultrasound image using a multi-layer neural network comprising at least one layer selected from the group consisting of: a pooling layer, a rectified linear units (ReLU) layer, a convolution layer, a dense layer, a pad layer, a concatenate layer, and an upscale layer.

17. The method of claim 12 , wherein determining whether the ultrasound image contains the target anatomical view comprises:
identifying an anatomical view contained in the ultrasound image using the automated image processing technique; and
determining whether the anatomical view contained in the ultrasound image matches the target anatomical view.

18. The method of claim 17 , further comprising:
responsive to determining that the anatomical view contained in the ultrasound image does not match the target anatomical view, generating the at least one instruction using the anatomical view contained in the ultrasound image.

19. The method of claim 12 , wherein providing the at least one instruction comprises providing an instruction to move the ultrasound device in a translational direction and/or a rotational direction.

20. The method of claim 12 , wherein providing the at least one instruction to the operator comprises providing the at least one instruction to the subject.

21. A system, comprising:
an ultrasound device configured to capture an ultrasound image of a subject; and
a computing device communicatively coupled to the

Assignees

• Butterfly Network Inc

Inventors

• Nouri Daniel
• Rothberg Alex
• De Jonge Matthew
• Jia Jimmy
• Rothberg Jonathan M
• Sofka Michal
• Neben Abraham

Classifications

• A61B8/46 Primary

  Ultrasonic, sonic or infrasonic diagnostic devices with special arrangements for interfacing with the operator or the patient · CPC title

• G06V30/19173

  Classification techniques · CPC title

• G06V10/82 Primary

  using neural networks · CPC title

• G06F18/24133 Primary

  Distances to prototypes · CPC title

• G06V10/454

  Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN] · CPC title