Augmented reality system for efficient and intuitive document classification

Patent metadata

Field	Value
Publication number	US-10831280-B2
Application number	US-201816155395-A
Country	US
Kind code	B2
Filing date	Oct 9, 2018
Priority date	Oct 9, 2018
Publication date	Nov 10, 2020
Grant date	Nov 10, 2020

Abstract

A computer-implemented method of using augmented reality (AR) to classify documents includes displaying one or more visual representations of one or more actions of a mobile device application in an AR display of a mobile device, selecting, by the mobile device, one of the one or more visual representations of one of the one or more actions for a document associated with the mobile device application in response to the mobile device being gestured toward the one visual representation of the one action, and classifying the document by performing the selected one action on the document.

First claim

What is claimed is:

1. A computer-implemented method of using augmented reality (AR) to classify documents, comprising the steps of:
displaying one or more visual representations of one or more actions of a mobile device application in an AR display of a mobile device;
gesturing, by a user, the mobile device toward one of the one or more visual representations of one of the one or more actions for a document associated with the mobile device application;
selecting by the mobile device, the one action in response to the mobile device being gestured toward the one visual representation of the one action; and
classifying the document by performing the selected one action on the document.

2. The method of claim 1 , wherein selecting, by the mobile device, one of the one or more visual representations of one of the one or more actions for a document associated with the mobile device application comprises
using a gyroscope and an accelerometer associated with the mobile device to match the gesture of the mobile device being gestured with the one visual representation of the one action.

3. The method of claim 1 , wherein the AR display is one of a head mounted display or a smart glass on the mobile device.

4. The method of claim 1 , farther comprising:
determining the one or more actions of the mobile device application; and
contextually labeling the one or more visual representations of the one or more actions based on their contents, wherein the contents of a visual representation of an action include those documents classified by the action associated with the visual representation.

5. The method of claim 4 , wherein determining the one or more actions of the mobile device application further comprises
presenting a plurality of AR objects in the AR display,
presenting a list of the one or more actions of the mobile device application in the AR display,
gesturing, by the user, the mobile device toward an action visible in the AR display;
scrolling through the list of one or more actions wherein the action in the list of one or more actions is selected in response to the mobile device being gestured toward the action; and
associating the selected in the list of one or more actions with one of the plurality of AR objects, in response to the mobile device being gestured from the selected action to the one AR object, wherein the one AR object becomes the visual representation of the selected action.

6. The method of claim 4 , wherein determining the one or more actions of the mobile device application further comprises
determining, by the mobile device, a predetermined number of most used actions of the mobile device application, and
associating, by the mobile device, each of the most used actions with one of a plurality of AR objects that are visible in the AR display, wherein the plurality of AR objects become the visual representations of the most used actions.

7. A system for using augmented reality (AR) to classify documents, comprising:
an AR display device; and
a mobile computer system operatively coupled to the AR display device that includes at least one processor tangibly embodying a program of instructions executed by the mobile computer system to cause the mobile computer system to perform a method comprising the steps of:
receiving a document associated with a mobile computer system application;
gesturing, by a user, the mobile computer system toward one of one or more visual representations of one of one or more actions of the mobile compute system application for the received document;
selecting by the mobile computer system, the one action in response to the mobile computer system being gestured toward the visual representations of the one action; and
classifying the received document by performing the selected one action on the document.

8. The system of claim 7 , wherein selecting, by the mobile computer system, one of one or more visual representations of one of one or more actions of the mobile computer system application for the received document comprises
using a gyroscope and an accelerometer associated with the mobile computer system being gestured to match the gesture with the one visual representation of the one action.

9. The system of claim 7 , wherein the AR display device is one of a head mounted display or a smart glass on the mobile computer system.

10. The system of claim 7 , wherein the method further comprises:
determining the one or more actions of the mobile computer system application;
displaying the one or more visual representations of the one or more actions of the mobile computer system application in the AR display device of the mobile computer system; and
contextually labeling the one or more visual representations of the one or more actions based on their contents, wherein the contents of a visual representation of an action include those documents classified by the action associated with the visual representation.

11. The system of claim 10 , wherein determining the one or more actions of the mobile computer system application further comprises
presenting a plurality of AR objects in the AR display device,
presenting a list of the one or more actions of the mobile computer system application in the AR display device,
gesturing, by the user, the mobile device toward an action visible in the AR display;
scrolling through the list of one or more actions wherein the action in the list of one or more actions is selected in response to the mobile computer system being gestured toward the action; and
associating the selected action in the list of one or more actions with one of the plurality of AR objects, in response to the mobile computer system being gestured from the selected action to the one AR object, wherein the one AR object becomes the visual representation of the selected action.

12. The system of claim 10 wherein determining the one or more actions of the mobile computer system application further comprises determining, by the mobile computer system, a predetermined number of most used actions of the mobile computer system application, and associating, by the mobile computer system, each of the most used actions with one of a plurality of AR objects that are visible in the AR display device, wherein the plurality of AR objects become the visual representations of the most used actions.

13. A computer program product for using augmented reality (AR) to classify documents, comprising a non-transitory program storage device readable by a mobile computer, tangibly embodying a program of instructions executed by the mobile computer to cause the mobile computer to perform a method comprising the steps of:
gesturing, by a user, the mobile computer toward one of the one or more visual representations of one of the one or more actions of a mobile computer application for a document associated with the mobile device application;
selecting, by the mobile computer, the one action in response to the mobile device being gestured toward one of one or more visual representations of the one action, wherein the mobile computer uses a gyroscope and an accelerometer associated with the mobile device being gestured to match the gesture with the one visual representation of the one action; and
classifying the document by performing the selected one action on the one document.

14. The computer program product of claim 13 , wherein the method further comprises displaying the one or more visual representations of the one or more actions of the mobile computer application in an AR display of the mobile computer.

15. The computer program product of claim 14 w

Assignees

• IBM

Inventors

• Andres Josh
• Eberbach Adam
• Mehedy Lenin
• Ting Yi Wei

Classifications

• G06F3/04815 Primary

  Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object · CPC title

• G06V40/28

  Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113) · CPC title

• G06V20/20

  in augmented reality scenes · CPC title

• G06V30/40

  Document-oriented image-based pattern recognition · CPC title

• G06F3/017 Primary

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title