Method for performing and visualizing analytics operations over data using augmented reality

Patent metadata

Field	Value
Publication number	US-11176377-B2
Application number	US-202017015276-A
Country	US
Kind code	B2
Filing date	Sep 9, 2020
Priority date	Oct 22, 2018
Publication date	Nov 16, 2021
Grant date	Nov 16, 2021

Abstract

The exemplified system and method facilitates process, grammar, and framework to perform analytics operations, and visualize the result of analytics operations using augmented reality. The exemplified system and method can be used, but is not limited to, for augmented reality presentations of physical objects as paper documents, digital or printed signage, posters, physical or digital displays, real-world objects, indoor and outdoor spaces, hardware device displays, vehicle dashboards, and other real-world scenes.

First claim

What is claimed is:

1. An augmented reality system comprising:
one or more processors; and
memory having instructions stored thereon, wherein execution of the instructions by the one or more processors cause the one or more processors to:
receive a video stream from a video capture sensor;
determine a plurality of structured data elements in one or more frames of the video stream, including a first set of structured data elements in a first set of one or more frames according to a data model;
parameterize the first set of structured data elements to a pre-defined encoding space, wherein the parameterized first set of structured data elements are presented via a pre-defined presentation model; and
present, via a graphical user interface, a first visual output corresponding to the video stream and a second visual output having a plurality of augmented data elements associated with the determined plurality of structured data elements.

2. The system of claim 1 , wherein the parameterized first set of structured data elements are presented via a pre-defined presentation model by encoding the parameterized first set of structured data elements to a pre-defined grounding space, wherein the encoding space is independent of the grounding space.

3. The system of claim 2 , wherein the command is selected from the group consisting of a grouping command, an exclude command, an append row command, an associate command, an append and aggregate command, and a sorting command.

4. The system of claim 3 , wherein the instructions to detect the data elements in the one or more frames comprises instructions that when executed by the one or more processors, cause the one or more processors to detect at least one of parallel lines, straight lines, and co-aligned text groups in a frame of the first set one or more frames.

5. The system of claim 1 , wherein the instructions to determine the first set of structured data elements comprise instructions that when executed by the one or more processors, cause the one or more processors to:
receive, via the graphical user interface, a gesture input corresponding to an augmented data element of the plurality of augmented data elements;
determine a command based on the received gesture input; and
execute the command.

6. The system of claim 1 , wherein the instructions to determine the first set of structured data elements comprise instructions that when executed by the one or more processors, cause the one or more processors to:
detect data elements in the first set of one or more frames;
segment the data elements from the first set of one or more frames; and
perform optical character recognition operation of the segmented data elements to generate a plurality of arrayed text data.

7. The system of claim 1 , wherein the instructions, when executed by the one or more processors, further cause the one or more processors to:
determine a second set of structured data elements in the first set of one or more frames, wherein the second set of structured data elements correspond to a data type, and wherein the first set of structured data elements correspond to an attribute type, and;
receive, via the graphical user interface, a first gesture query input corresponding to a first augmented data element of the plurality of augmented data elements, wherein the first augmented data element is associated with the first set of structured data elements corresponding to the attribute type;
determine, one or more structured data elements of the second set of structured data elements associated with the first augmented data element;
render the one or more structured data elements of the second set of structured data elements as an augmented object associated with the first augmented data element; and
present, via the graphical user interface, the rendered augmented object of the one or more structured data elements of the second set of structured data elements.

8. The system of claim 1 , wherein the instructions, when executed by the one or more processors, further cause the one or more processors to:
generate a result of a mathematical operator performed on values derived from the first structured data elements; and
present, via the graphical user interface, the result of the mathematical operator.

9. The system of claim 8 , wherein the instructions, when executed by the one or more processors, further cause the one or more processors to:
generate a second result of a second mathematical operator performed on second set of values derived from the second structured data elements; and
co-present, via the graphical user interface, the second result of the second mathematical operator next to the result of the mathematical operator.

10. The system of claim 8 , wherein the mathematical operator is selected from group consisting of a maximum operator, a minimum operator, an average (mean) operator, and a mode operator.

11. The system of claim 1 , wherein the instructions, when executed by the one or more processors, further cause the one or more processors to:
find contours of table cells in the one or more frames;
sub-select a plurality of corner elements of the table cells using an optical flow tracking operation; and
update rendered augmented object of the one or more structured data elements based on a grounding space derived from the optical flow tracking operation.

12. The system of claim 1 , wherein the instructions, when executed by the one or more processors, further cause the one or more processors to:
evaluate a plurality of interactive visual querying grammar in the first set of structured data elements; and
execute an associated operator corresponding to a determined interactive visual querying grammar.

13. The system of claim 1 , wherein the second visual output having the plurality of augmented data elements are presented as augmented highlights selected from the group consisting of occlusion masks, colors, and text annotations.

14. The system of claim 1 , further comprising a network interface, wherein the system is configured as a smart phone, laptop, or tablet.

15. The system of claim 1 , further comprising a head set assembly, wherein the system is configured as a headset or AR wearable device.

16. A non-transitory computer readable medium comprising having instructions stored thereon, wherein execution of the instructions by a one or more processors of a computing system cause the one or more processors to:
receive a video stream from a video capture sensor of the computing system;
determine a plurality of structured data elements in one or more frames of the video stream, including a first set of structured data elements in a first set of one or more frames according to a data model;
parameterize the first set of structured data elements to a pre-defined encoding space, wherein the parameterized first set of structured data elements are presented via a pre-defined presentation model; and
present, via a graphical user interface, a first visual output corresponding to the video stream and a second visual output having a plurality of augmented data elements associated with the determined plurality of structured data elements.

17. The computer readable medium of claim 16 , wherein the parameterized first set of structured data elements are presented via a pre-defined presentation model by encoding the parameterized first set of structured data elements to a pre-defined grounding space, wherein the encoding space is independent of the grounding space.

18. The computer readable medium of claim 16 , wherein

Assignees

• Ohio State Innovation Foundation

Inventors

• Nandi Arnab
• Burley Codi

Classifications

• G06V40/20

  Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16) · CPC title

• G06F3/04883 Primary

  for inputting data by handwriting, e.g. gesture or text · CPC title

• G06V20/20

  in augmented reality scenes · CPC title

• G06V30/412

  Layout analysis of documents structured with printed lines or input boxes, e.g. business forms or tables · CPC title

• G06T2207/10016

  Video; Image sequence · CPC title