Augmented reality interface for assisting a user to operate an ultrasound device

Patent metadata

Field	Value
Publication number	US-11185307-B2
Application number	US-202016889944-A
Country	US
Kind code	B2
Filing date	Jun 2, 2020
Priority date	Jun 20, 2016
Publication date	Nov 30, 2021
Grant date	Nov 30, 2021

Abstract

Aspects of the technology described herein relate to techniques for guiding an operator to use an ultrasound device. Thereby, operators with little or no experience operating ultrasound devices may capture medically relevant ultrasound images and/or interpret the contents of the obtained ultrasound images. For example, some of the techniques disclosed herein may be used to identify a particular anatomical view of a subject to image with an ultrasound device, guide an operator of the ultrasound device to capture an ultrasound image of the subject that contains the particular anatomical view, and/or analyze the captured ultrasound image to identify medical information about the subject.

First claim

What is claimed is:

1. A mobile device, comprising:
a camera;
a display; and
at least one processor configured to:
identify a pose of an ultrasound device based at least in part on a non-acoustic image captured with the camera of the mobile device, wherein the non-acoustic image depicts the ultrasound device and a subject being imaged by the ultrasound device;
generate an augmented reality interface, the augmented reality interface comprising an ultrasound image captured by the ultrasound device and overlaid onto the non-acoustic image; and
present the augmented reality interface on the display of the mobile device;

wherein:
the at least one processor is configured, when generating the augmented reality interface, to overlay the ultrasound image onto a portion of the subject that is being imaged by the ultrasound device as depicted by the non-acoustic image;
the at least one processor is configured, when overlaying the ultrasound image onto the portion of the subject that is being imaged by the ultrasound device as depicted by the non-acoustic image, to position and orient the ultrasound image such that the ultrasound image extends from the ultrasound device into the portion of the subject that is being imaged by the ultrasound device as depicted by the non-acoustic image; and
the at least one processor is configured, when positioning and orienting the ultrasound image such that the ultrasound image extends from the ultrasound device into the portion of the subject that is being imaged by the ultrasound device as depicted by the non-acoustic image, to position and orient the ultrasound image such that the ultrasound image extends from an end of the ultrasound device in contact with the subject into the portion of the subject that is being imaged by the ultrasound device as depicted by the non-acoustic image.

2. The mobile device of claim 1 , wherein the at least one processor is configured, when identifying the pose of the ultrasound device, to identify a distinctive characteristic of the ultrasound device as depicted by the non-acoustic image.

3. The mobile device of claim 1 , wherein the at least one processor is configured, when identifying the pose of the ultrasound device, to use deep learning techniques.

4. The mobile device of claim 1 , wherein the camera comprises a front-facing camera.

5. A method, comprising:
identifying a pose of an ultrasound device based at least in part on a non-acoustic image captured with a camera of a mobile device, wherein the non-acoustic image depicts the ultrasound device and a subject being imaged by the ultrasound device;
generating an augmented reality interface, the augmented reality interface comprising an ultrasound image captured by the ultrasound device and overlaid onto the non-acoustic image; and
presenting the augmented reality interface on a display of the mobile device;
wherein:
generating the augmented reality interface comprises overlaying the ultrasound image onto a portion of the subject that is being imaged by the ultrasound device as depicted by the non-acoustic image;
overlaying the ultrasound image onto the portion of the subject that is being imaged by the ultrasound device as depicted by the non-acoustic image comprises positioning and orienting the ultrasound image such that the ultrasound image extends from the ultrasound device into the portion of the subject that is being imaged by the ultrasound device as depicted by the non-acoustic image; and
positioning and orienting the ultrasound image such that the ultrasound image extends from the ultrasound device into the portion of the subject that is being imaged by the ultrasound device as depicted by the non-acoustic image comprises positioning and orienting the ultrasound image such that the ultrasound image extends from an end of the ultrasound device in contact with the subject into the portion of the subject that is being imaged by the ultrasound device as depicted by the non-acoustic image.

6. The method of claim 5 , wherein identifying the pose of the ultrasound device comprises identifying a distinctive characteristic of the ultrasound device as depicted by the non-acoustic image.

7. The method of claim 5 , wherein identifying the pose of the ultrasound device comprises using deep learning techniques.

8. The method of claim 5 , wherein the camera comprises a front-facing camera.

9. At least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by at least one processor of a mobile device, cause the at least one processor to:
identify a pose of an ultrasound device based at least in part on a non-acoustic image captured with a camera of the mobile device, wherein the non-acoustic image depicts the ultrasound device and a subject being imaged by the ultrasound device;
generate an augmented reality interface, the augmented reality interface comprising an ultrasound image captured by the ultrasound device and overlaid onto the non-acoustic image; and
present the augmented reality interface on a display of the mobile device;
wherein:
the instructions, when executed by the at least one processor of the mobile device, cause the at least one processor to generate the augmented reality interface by overlaying the ultrasound image onto a portion of the subject that is being imaged by the ultrasound device as depicted by the non-acoustic image; and
overlaying the ultrasound image onto the portion of the subject that is being imaged by the ultrasound device as depicted by the non-acoustic image comprises positioning and orienting the ultrasound image such that the ultrasound image extends from an end of the ultrasound device in contact with the subject into the portion of the subject that is being imaged by the ultrasound device as depicted by the non-acoustic image.

10. The at least one non-transitory computer-readable storage medium of claim 9 , wherein the instructions, when executed by the at least one processor of the mobile device, cause the at least one processor to identify the pose of the ultrasound device by identifying a distinctive characteristic of the ultrasound device as depicted by the non-acoustic image.

11. The at least one non-transitory computer-readable storage medium of claim 9 , wherein the instructions, when executed by the at least one processor of the mobile device, cause the at least one processor to identify the pose of the ultrasound device using deep learning techniques.

12. The at least one non-transitory computer-readable storage medium of claim 9 , wherein the camera comprises a front-facing camera.

Assignees

• Bfly Operations Inc

Inventors

• De Jonge Matthew
• Schneider Robert
• Elgena David
• Rothberg Alex
• Rothberg Jonathan M
• Sofka Michal
• Gafner Tomer
• Thiele Karl
• Neben Abraham

Classifications

• A61B8/46 Primary

  Ultrasonic, sonic or infrasonic diagnostic devices with special arrangements for interfacing with the operator or the patient · CPC title

• G06V30/19173

  Classification techniques · CPC title

• G06V10/82 Primary

  using neural networks · CPC title

• G06F18/24133 Primary

  Distances to prototypes · CPC title

• G06V10/454

  Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN] · CPC title