Systems and methods for an augmented display system

Patent metadata

Field	Value
Publication number	US-10535198-B2
Application number	US-201816003867-A
Country	US
Kind code	B2
Filing date	Jun 8, 2018
Priority date	Jun 9, 2017
Publication date	Jan 14, 2020
Grant date	Jan 14, 2020

Abstract

Described in detail herein are systems and methods for an augmented display system. A computing system can determine whether a physical object is absent from a designated location in a facility in response to data retrieved from the database. The computing system can insert a virtual element in a planogram corresponding to the designated location of the facility in response to determining the physical object is absent. The computing system can augment via interaction with the application executing on the portable electronic device.

First claim

The invention claimed is:

1. An augmented display system, the system comprising:
one or more data storage devices storing at least one database and a planogram;
one or more servers coupled to the one or more data storage devices, configured to:
determine a physical object is absent from a designated location in a facility in response to data retrieved from the database;
inserting a virtual element in a planogram corresponding to the designated location of the facility in response to determining the physical object is absent;
establishing a session with a portable electronic device in response to a request received from an application executing on the portable electronic and a determination that the portable electronic device is in the facility;
receive a discrete element in a physical scene from the portable electronic device, the physical scene corresponding to a field of view of an image capturing device of the portable electronic device and being parsed to extract the discrete virtual element from the scene, the discrete virtual element corresponding to a label;
extract the string of alphanumeric characters from the discrete element;
determine whether the string corresponds to the physical object;
augment, via interaction with the application executing on the portable electronic device, the scene rendered on a touch-sensitive display of the portable device to insert the virtual element associated with the designated location of the physical object in the planogram into the scene adjacent to the label; and
update the database based on a user gesture on the touch-sensitive display in response to the virtual element being included in the physical scene.

2. The system of claim 1 , wherein the portable electronic device is configured to:
execute an application stored in memory via the processor;
control the operation of the image capturing device, in response to executing the application to contemporaneously and continuously image an area within a field of view of the image capturing device;
render on the touch-sensitive display the physical scene within the field of view of the image capturing device;
parse the physical scene rendered on the touch-sensitive display into the discrete elements based on dimensions of items in the physical scene;
render the virtual element associated with the physical objects on the touch touch-sensitive display;
receive the user gesture associated with the virtual element, via the touch sensitive display; and
transmit the user gesture to the one or more servers.

3. The system of claim 1 , wherein the one or more servers are configured to: determine the physical object is absent from the designated location based on a quantity of each set of like physical object stored in the database, being less than a specified amount.

4. The system of claim 1 , wherein the application is associated with a user of the portable electronic device.

5. The system of claim 4 , wherein the one or more servers are configured to:
trigger an alert in response to updating the database based on a user gesture on the touch-sensitive display in response to the virtual element being included in the physical scene.

6. The system of claim 5 , wherein the one or more servers are configured to:
update the database to indicate the user of the portable device transmitted the user gesture confirming the physical objects are absent from the designated location;
generate a score associated with the user of the portable electronic device; and
transmit the score to the portable electronic device.

7. The system of claim 6 , wherein the portable electronic device is configured to:
render instructions on the touch-sensitive display instructing the user of the portable electronic device to navigate to a location in the facility associated with additional physical objects.

8. The system of claim 1 , wherein the virtual element is an image including a selectable link.

9. The system of claim 8 , wherein the selectable link includes information associated the physical object.

10. The system of claim 1 , wherein the one or more servers are configured to:
receive a captured image of the physical scene in response to the user gesture on the touch-sensitive screen.

11. An augmented display method, the method comprising:
determining, via one or more servers, whether a physical object is absent from a designated location in a facility in response to data retrieved from the database;
inserting, via the one or more servers, a virtual element in a planogram corresponding to the designated location of the facility in response to determining the physical object is absent;
establishing, via the one or more servers, a session with a portable electronic device in response to a request received from an application executing on the portable electronic and a determination that the portable electronic device is in the facility;
receiving, via the one or more servers, a discrete element in a physical scene from the portable electronic device, the physical scene corresponding to a field of view of an image capturing device of the portable electronic device and being parsed to extract the discrete virtual element from the scene, the discrete virtual element corresponding to a label;
extracting, via the one or more servers, the string of alphanumeric characters from the discrete element;
determining, via the one or more servers, whether the string corresponds to the physical object;
augmenting, via the one or more servers, via interaction with the application executing on the portable electronic device, the scene rendered on a touch-sensitive display of the portable device to insert the virtual element associated with the designated location of the physical object in the planogram into the scene adjacent to the label; and
updating, via the one or more servers, the database based on a user gesture on the touch-sensitive display in response to the virtual element being included in the physical scene.

12. The method of claim 11 , further comprising:
executing, via the portable electronic device, an application stored in memory via the processor;
controlling, via the portable electronic device, the operation of the image capturing device, in response to executing the application to contemporaneously and continuously image an area within a field of view of the image capturing device;
rendering, via the portable electronic device, on the touch-sensitive display the physical scene within the field of view of the image capturing device;
parsing, via the portable electronic device, the physical scene rendered on the touch-sensitive display into the discrete elements based on dimensions of items in the physical scene;
rendering, via the portable electronic device, the virtual element associated with the physical objects on the touch touch-sensitive display;
receiving, via the portable electronic device, the user gesture associated with the virtual element, via the touch sensitive display; and
transmitting, via the portable electronic device, the user gestures to the one or more servers.

13. The method of claim 11 , further comprising determining, via the one or more servers, the physical object is absent from the designated location based on a quantity of each set of like physical object stored in the database, being less than a specified amount.

14. The method of claim 11 , wherein the application is associated with a user of the portable electronic device.

15. The method of claim 14 , further comprising:
triggering, via the one or more servers, an alert in response to updating the database based on a user gestu

Assignees

• Walmart Apollo Llc

Inventors

• Enssle Benjamin D
• Brightwell David Blair
• Brooks Cristy Crane
• Bryan Greg A
• George Sonney
• Freeman Barrett
• Padwal Sachin

Classifications

• H04N23/60

  Control of cameras or camera modules · CPC title

• H04N23/62

  Control of parameters via user interfaces · CPC title

• G06Q30/0202

  Market predictions or forecasting for commercial activities · CPC title

• G06F3/0488

  using a touch-screen or digitiser, e.g. input of commands through traced gestures · CPC title

• G06F3/011

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title