Augmented reality interface for assisting a user to operate an ultrasound device

Patent metadata

Field	Value
Publication number	US-10856848-B2
Application number	US-201715626885-A
Country	US
Kind code	B2
Filing date	Jun 19, 2017
Priority date	Jun 20, 2016
Publication date	Dec 8, 2020
Grant date	Dec 8, 2020

Abstract

Aspects of the technology described herein relate to techniques for guiding an operator to use an ultrasound device. Thereby, operators with little or no experience operating ultrasound devices may capture medically relevant ultrasound images and/or interpret the contents of the obtained ultrasound images. For example, some of the techniques disclosed herein may be used to identify a particular anatomical view of a subject to image with an ultrasound device, guide an operator of the ultrasound device to capture an ultrasound image of the subject that contains the particular anatomical view, and/or analyze the captured ultrasound image to identify medical information about the subject.

First claim

What is claimed is:

1. A method for guiding an operator of an ultrasound device in positioning the ultrasound device, the method comprising:
using a single mobile device, separate from the ultrasound device and comprising at least one processor, a camera, and a display, to perform:
automatically identifying a pose of the ultrasound device at least in part by imaging the ultrasound device with the camera of the single mobile device;
generating, using the single mobile device, a composite image of the ultrasound device and an indication for how to move the ultrasound device, the generating comprising overlaying the indication on an image of the ultrasound device at a location in the image determined using the identified pose of the ultrasound device; and
presenting the composite image to the operator of the ultrasound device using the display of the single mobile device.

2. The method of claim 1 , wherein automatically identifying the pose of the ultrasound device comprises identifying a location, in the image, of a marker on the ultrasound device.

3. The method of claim 2 , wherein automatically identifying the pose of the ultrasound device comprises identifying a position of the ultrasound device in the image using the identified location of the marker in the image.

4. The method of claim 1 , wherein the ultrasound device comprises at least one sensor configured to detect movement of the ultrasound device, and wherein automatically identifying the pose of the ultrasound device is performed using data collected by the at least one sensor.

5. The method of claim 4 , wherein the at least one sensor comprises a gyroscope.

6. The method of claim 4 , wherein the at least one sensor comprises an accelerometer.

7. The method of claim 1 , further comprising:
obtaining an ultrasound image captured by the ultrasound device; and
generating the indication using the ultrasound image.

8. The method of claim 7 , further comprising:
presenting the ultrasound image to the operator of the ultrasound device using the display of the mobile device.

9. A system for guiding an operator of an ultrasound device in positioning the ultrasound device, the system comprising:
a single mobile device, separate from the ultrasound device and comprising at least one processor, a camera, and a display, the single mobile device configured to:
automatically identify a pose of the ultrasound device at least in part by imaging the ultrasound device with the camera of the single mobile device;
generate, using the single mobile device, a composite image of the ultrasound device and an indication for how to move the ultrasound device, the generating comprising overlaying the indication on an image of the ultrasound device at a location in the image determined using the identified pose of the ultrasound device; and
present the composite image to the operator of the ultrasound device using the display of the single mobile device.

10. The system of claim 9 , further comprising:
the ultrasound device.

11. The system of claim 10 , wherein the ultrasound device comprises at least one sensor configured to detect movement of the ultrasound device.

12. The system of claim 9 , wherein the single mobile device is configured to automatically identify the pose of the ultrasound device at least in part by identifying a location, in the image, of a marker on the ultrasound device.

13. The system of claim 12 , wherein the single mobile device is configured to automatically identify the pose of the ultrasound device at least in part by identifying a position of the ultrasound device in the image using the identified location of the marker in the image.

14. The system of claim 11 , wherein the single mobile device is configured to automatically identify the pose of the ultrasound device at least in part by using data collected by the at least one sensor.

15. The system of claim 14 , wherein the at least one sensor comprises a gyroscope.

16. The system of claim 14 , wherein the at least one sensor comprises an accelerometer.

17. At least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by a single mobile device, the single mobile device comprising at least one processor, a camera, and a display, cause the single mobile device to:
automatically identify a pose of an ultrasound device at least in part by imaging the ultrasound device with the camera of the single mobile device;
generate, using the single mobile device, a composite image of the ultrasound device and an indication for how to move the ultrasound device, the generating comprising overlaying the indication on an image of the ultrasound device at a location in the image determined using the identified pose of the ultrasound device; and
present the composite image to an operator of the ultrasound device using the display of the single mobile device.

18. The method of claim 1 , wherein overlaying the indication on the image of the ultrasound device comprises overlaying an indication to move the ultrasound device in a specific direction.

19. The method of claim 18 , wherein overlaying the indication to move the ultrasound device in the specific direction comprises overlaying, on the image of the ultrasound device, an arrow pointing in the direction.

20. The method of claim 1 , wherein overlaying the indication on the image of the ultrasound device comprises overlaying an indication to rotate the ultrasound device.

21. The system of claim 9 , wherein overlaying the indication on the image of the ultrasound device comprises overlaying an indication to move the ultrasound device in a specific direction.

22. The system of claim 9 , wherein overlaying the indication on the image of the ultrasound device comprises overlaying an indication to rotate the ultrasound device.

23. The at least one non-transitory computer-readable storage medium of claim 17 , wherein overlaying the indication on the image of the ultrasound device comprises overlaying an indication to move the ultrasound device in a specific direction.

24. The at least one non-transitory computer-readable storage medium of claim 17 , wherein overlaying the indication on the image of the ultrasound device comprises overlaying an indication to rotate the ultrasound device.

25. The at least one non-transitory computer-readable storage medium of claim 17 , wherein the single mobile device is configured to automatically identify the pose of the ultrasound device at least in part by identifying a location, in the image, of a marker on the ultrasound device.

26. The at least one non-transitory computer-readable storage medium of claim 25 , wherein the single mobile device is configured to automatically identify the pose of the ultrasound device at least in part by identifying a position of the ultrasound device in the image using the identified location of the marker in the image.

27. The at least one non-transitory computer-readable storage medium of claim 17 , wherein the ultrasound device comprises at least one sensor configured to detect movement of the ultrasound device, wherein the single mobile device is configured to automatically identify the pose of the ultrasound device at least in part by using data collected by the at least one sensor.

28. The at least one non-transitory computer-readable storage medium of claim 27 , wherein

Assignees

• Butterfly Network Inc

Inventors

• Gafner Tomer
• De Jonge Matthew
• Schneider Robert
• Elgena David
• Rothberg Alex
• Rothberg Jonathan M
• Sofka Michal
• Thiele Karl
• Neben Abraham

Classifications

• A61B8/46 Primary

  Ultrasonic, sonic or infrasonic diagnostic devices with special arrangements for interfacing with the operator or the patient · CPC title

• G06V30/19173

  Classification techniques · CPC title

• G06V10/82 Primary

  using neural networks · CPC title

• G06F18/24133 Primary

  Distances to prototypes · CPC title

• G06V10/454

  Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN] · CPC title