Methods and systems for image processing to present data in augmented reality

Patent metadata

Field	Value
Publication number	US-11727683-B2
Application number	US-202217850263-A
Country	US
Kind code	B2
Filing date	Jun 27, 2022
Priority date	Mar 18, 2019
Publication date	Aug 15, 2023
Grant date	Aug 15, 2023

Abstract

Methods and systems for displaying information in augmented reality are described. In one embodiment, a system comprises a camera, a memory, a display, and a processor configured to (a) detect a first prescription drug in a field of view of a camera by detecting a glyph printed on a label on a container of the first prescription drug, (b) determine a first identifier associated with the first prescription drug using optical character recognition, (c) determine whether a user has access to additional data about the first prescription drug, (d) request the additional data about the first prescription drug from a database, and (e) display the additional data in augmented reality adjacent to the first prescription drug on the display such that the additional data moves when the field of view of the camera changes such that the additional data remains displayed adjacent to the first prescription drug.

First claim

We claim:

1. A system comprising:
a camera;
a memory component;
a display communicatively coupled to the camera, the display configured to receive and present a live stream of images from within a field of view of the camera; and
a processor communicatively coupled to the camera, the memory component, and the display, the processor configured to:
(a) detect a glyph in the live stream of the images by the processor performing image analysis on the images;
(b) use the glyph to identify a current location of a container within the live stream of the images, wherein the glyph is contained on a sub region of the container;
(c) determine a first identifier associated with the container, by:
unwarping the live stream of the images;
in response to completing unwarping the live stream of the images, performing optical character recognition on at least one of the images from the camera; and
determining the first identifier based on the optical character recognition;

(d) determine whether a user has access to additional data about the container; and
(e) request the additional data about the container from a database; and

an augmented reality subsystem communicatively coupled to the camera, the memory component, the display, and the processor, the augmented reality subsystem configured to:
display the additional data about the container in an augmented reality foreground image, the additional data that is displayed including interactive buttons on the display, the interactive buttons configured to be activated to (i) transmit a request to refill a drug, (ii) call a doctor that prescribed the drug, and (iii) display drug interaction data about the drug, the augmented reality foreground image being displayed over the live stream of the images such that the additional data is displayed at a relative location adjacent to the current location and maintains the relative location as the current location changes in response to movement of the container or changes in the field of view of the camera.

2. A method comprising:
receiving, from a camera, a live stream of images from within a field of view of the camera;
performing, by at least one processor, image analysis on the live stream of the images;
detecting, by the at least one processor, a glyph in at least one of the live stream of the images as an output of the image analysis;
identifying a current location of a first prescription drug in the live stream of the images, using the glyph, by the at least one processor;
determining, by the at least one processor, a first identifier associated with the first prescription drug, by:
unwarping the live stream of the images to generate unwarped images for detecting the first identifier printed on a rounded surface of a pill bottle; and
using optical character recognition on at least one of the unwarped images to determine the first identifier associated with the first prescription drug;

determining whether a user has access to additional data about the first prescription drug, by the at least one processor;
requesting the additional data about the first prescription drug from a database, by the at least one processor; and
displaying the additional data about the first prescription drug in an augmented reality foreground image, by the at least one processor, the additional data that is displayed including interactive buttons on the display, the interactive buttons configured to be activated to (i) transmit a request to refill the first prescription drug, (ii) call a doctor that prescribed the first prescription drug, and (iii) display drug interaction data about the first prescription drug, the augmented reality foreground image being displayed over the live stream of the images such that the additional data is displayed at a relative location adjacent to the current location of the first prescription drug and maintains the relative location as the current location changes in response to movement of the first prescription drug or changes in the field of view of the camera.

3. The method of claim 2 , wherein detecting the first prescription drug in the live stream of the images further comprises:
detecting, by the at least one processor, the glyph printed on a label on a container of the first prescription drug.

4. The method of claim 2 , wherein using the glyph in the live stream of the images to identify the current location of a first prescription drug in the live stream of the images further comprises:
determining, by the at least one processor, whether an object in the live stream of the images matches a 3D model of the first prescription drug.

5. The method of claim 2 , wherein determining whether the user has access to the additional data about the first prescription drug further comprises:
determining, by the at least one processor, whether valid user credentials have been entered into an application used to display the additional data about the first prescription drug in augmented reality.

6. The method of claim 5 , further comprising:
displaying, by the at least one processor, an error message when the valid user credentials have not been entered into the application.

7. The method of claim 2 , wherein the first identifier is a prescription number.

8. The method of claim 2 , further comprising:
detecting, by the at least one processor, activation of the interactive buttons; and
performing, by the at least one processor, an action associated with the interactive buttons in response to activation of the interactive buttons, the action including transmitting the request to refill the first prescription drug, calling the doctor that prescribed the first prescription drug, and displaying the drug interaction data about the first prescription drug.

9. The method of claim 8 , wherein the action comprises transmitting the request to refill the first prescription drug to a server.

10. The method of claim 8 , wherein the action comprises invoking a cellular phone function to place a phone call to the doctor who prescribed the first prescription drug.

11. The method of claim 2 , wherein the additional data further comprises at least one from the group consisting of: a member name, a prescription number, a name of the first prescription drug, a quantity of the first prescription drug, a number of refills, a last fill date, a prescription expiration date, the doctor who prescribed the first prescription drug, and a phone number of the doctor who prescribed the first prescription drug.

12. The method of claim 2 , further comprising:
detecting a second substance in the live stream of the images, by the at least one processor;
determining a second identifier associated with the second substance, by the at least one processor;
transmitting a request for the drug interaction data between the first prescription drug and the second substance, by the at least one processor; and
displaying the drug interaction data in the augmented reality foreground image adjacent to the first prescription drug or the second substance, by the at least one processor.

13. The method of claim 2 , further comprising:
predicting, by the at least one processor, a number of doses remaining in a fill of the first prescription drug based on a current date, a last refill date, and dosing instructions; and
displaying the number of doses remaining in the fill in the augmented reality foreground image, by the at least one processor.

14. A non-transitory, machine-readable medium comprising instructions thereon, which, when executed by a processor, causes the processor to perform a method comprising:
receiving, from a camera, a live stream of images fr

Assignees

• Express Scripts Strategic Dev Inc

Inventors

• Mcdole Matt D
• Adhiappan Sivaram
• Gaddy Robert K
• Ruiz Carlos G
• Sanli Murat

Classifications

• G06V20/20 Primary

  in augmented reality scenes · CPC title

• G06K7/143

  Glyph-codes · CPC title

• G06K7/1443

  locating of the code in an image · CPC title

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G16H80/00

  ICT specially adapted for facilitating communication between medical practitioners or patients, e.g. for collaborative diagnosis, therapy or health monitoring · CPC title