Augmented reality interface for assisting a user to operate an ultrasound device

Patent metadata

Field	Value
Publication number	US-11564657-B2
Application number	US-202117330343-A
Country	US
Kind code	B2
Filing date	May 25, 2021
Priority date	Jun 20, 2016
Publication date	Jan 31, 2023
Grant date	Jan 31, 2023

Abstract

Aspects of the technology described herein relate to techniques for guiding an operator to use an ultrasound device. Thereby, operators with little or no experience operating ultrasound devices may capture medically relevant ultrasound images and/or interpret the contents of the obtained ultrasound images. For example, some of the techniques disclosed herein may be used to identify a particular anatomical view of a subject to image with an ultrasound device, guide an operator of the ultrasound device to capture an ultrasound image of the subject that contains the particular anatomical view, and/or analyze the captured ultrasound image to identify medical information about the subject.

First claim

What is claimed is:

1. A method, comprising:
capturing, using a camera coupled to a mobile device, an image of an ultrasound device being used by an operator when the ultrasound device is in the camera's field of view;
identifying a pose of the ultrasound device based at least in part on the image of the ultrasound device;
generating, using at least one processor of the mobile device, a composite image at least in part by overlaying, onto the image of the ultrasound device and using the pose of the ultrasound device, an indication of a direction in which the operator is to move the ultrasound device;
presenting the composite image to the operator via a display of the mobile device; and
prior to generating the composite image, using the mobile device to:
analyze an ultrasound image captured by the ultrasound device by using a trained model, and
determine the indication using an output of the trained model.

2. The method of claim 1 , wherein the ultrasound device has a marker disposed thereon, and wherein capturing the image of the ultrasound device comprises obtaining an image of the marker.

3. The method of claim 2 , wherein identifying the pose of the ultrasound device comprises identifying a location of the marker in the image of the ultrasound device.

4. The method of claim 1 , further comprising obtaining the ultrasound image captured by the ultrasound device.

5. The method of claim 4 , further comprising presenting the ultrasound image with the composite image via the display of the mobile device.

6. The method of claim 1 , wherein overlaying the indication of the direction in which the operator is to move the ultrasound device comprises overlaying an indication to rotate the ultrasound device.

7. The method of claim 1 , wherein overlaying the indication of the direction in which the operator is to move the ultrasound device comprises overlaying an indication to reposition the ultrasound device.

8. The method of claim 1 , wherein overlaying the indication of the direction in which the operator is to move the ultrasound device comprises overlaying an indication to move the ultrasound device up, down, left, or right.

9. A system, comprising:
a mobile device comprising a display and at least one processor configured to:
capture, using a camera coupled to the mobile device, an image of an ultrasound device being used by an operator when the ultrasound device is in the camera's field of view;
identify a pose of the ultrasound device based at least in part on the image of the ultrasound device;
generate, using the at least one processor of the mobile device, a composite image at least in part by overlaying, onto the image of the ultrasound device and using the pose of the ultrasound device, an indication of a direction in which the operator is to move the ultrasound device;
cause the display to present the composite image to the operator; and
prior to generating the composite image:
analyze an ultrasound image captured by the ultrasound device by using a trained model, and
determine the indication using an output of the trained model.

10. The system of claim 9 , wherein the ultrasound device comprises a marker disposed thereon, wherein the image of the ultrasound device comprises an image of the marker, and wherein the at least one processor is configured to identify the pose of the ultrasound device at least in part by identifying a location of the marker in the image of the ultrasound device.

11. The system of claim 10 , wherein the marker is selected from the group consisting of: a holographic marker, a dispersive marker, and an ArUco marker.

12. The system of claim 9 , further comprising the ultrasound device.

13. The system of claim 9 , wherein overlaying the indication of the direction in which the operator is to move the ultrasound device comprises overlaying an indication to rotate the ultrasound device.

14. The system of claim 9 , wherein overlaying the indication of the direction in which the operator is to move the ultrasound device comprises overlaying an indication to reposition the ultrasound device.

15. The system of claim 9 , wherein overlaying the indication of the direction in which the operator is to move the ultrasound device comprises overlaying an indication to move the ultrasound device up, down, left, or right.

16. At least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by at least one processor of a mobile device, cause the at least one processor to:
capture, using a camera coupled to the mobile device, an image of an ultrasound device being used by an operator when the ultrasound device is in the camera's field of view;
identify a pose of the ultrasound device based at least in part on the image of the ultrasound device;
generate, using the at least one processor of the mobile device, a composite image at least in part by overlaying, onto the image of the ultrasound device and using the pose of the ultrasound device, an indication of a direction in which the operator is to move the ultrasound device;
cause the display to present the composite image to the operator; and
prior to generating the composite image:
analyze an ultrasound image captured by the ultrasound device by using a trained model, and
determine the indication using an output of the trained model.

17. The at least one non-transitory computer-readable storage medium of claim 16 , wherein overlaying the indication of the direction in which the operator is to move the ultrasound device comprises overlaying an indication to rotate the ultrasound device.

18. The at least one non-transitory computer-readable storage medium of claim 16 , wherein overlaying the indication of the direction in which the operator is to move the ultrasound device comprises overlaying an indication to reposition the ultrasound device.

19. The at least one non-transitory computer-readable storage medium of claim 16 , wherein overlaying the indication of the direction in which the operator is to move the ultrasound device comprises overlaying an indication to move the ultrasound device up, down, left, or right.

Assignees

• Bfly Operations Inc

Inventors

• De Jonge Matthew
• Schneider Robert
• Elgena David
• Rothberg Alex
• Rothberg Jonathan M
• Sofka Michal
• Gafner Tomer
• Thiele Karl
• Neben Abraham

Classifications

• G06T2210/41

  Medical · CPC title

• G06T11/60

  Creating or editing images; Combining images with text · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• A61B2034/2065

  Tracking using image or pattern recognition · CPC title