AR glasses with event and user action control of external applications

Patent metadata

Field	Value
Publication number	US-10852540-B2
Application number	US-201816121901-A
Country	US
Kind code	B2
Filing date	Sep 5, 2018
Priority date	Feb 28, 2010
Publication date	Dec 1, 2020
Grant date	Dec 1, 2020

Abstract

This disclosure concerns an interactive head-mounted eyepiece with an integrated processor for handling content for display and an integrated image source for introducing the content to an optical assembly through which the user views a surrounding environment and the displayed content, wherein the eyepiece includes event and user action control of external applications.

First claim

What is claimed is:

1. A system comprising:
an interactive head-mounted device including an optical assembly configured to display virtual content and to enable viewing of at least a portion of a surrounding environment, an integrated processor for processing the virtual content for display, and an integrated image source for introducing the virtual content to the optical assembly;
a communications facility configured to connect the interactive head-mounted device to an external device; and
a user action capture device configured to detect a user action as input;
wherein the interactive head-mounted device is configured to detect, via a sensor of the interactive head-mounted device and independently of user input, a target that indicates that instructions are available to the user, and in response enable a command and control scheme for command and control of an external application resident on the external device, and
wherein the command and control scheme is configured to use user actions captured by the user action capture device as input to the external application, to send a request for the instructions to the external application in response to the user actions captured by the user action capture device, and to present the instructions to a user of the interactive head-mounted device responsive to the external application sending the instructions.

2. The system of claim 1 , wherein the sensor is a camera.

3. The system of claim 1 , wherein the external application stores the instructions, and wherein the instructions are provided to the user without receiving user input requesting the instructions.

4. The system of claim 1 , wherein the sensor is a position sensor configured to output coordinates of the interactive head-mounted device.

5. The system of claim 1 , wherein the user actions comprise one or more of user head movements, user eye movements, user voice commands, and user finger movements.

6. The system of claim 1 , wherein the external device comprises one or more of a computer, a smart phone, a storage-enabled device, and a communications system, and wherein the external device is a separate entity from the detected target.

7. The system of claim 1 , wherein the user action capture device comprises one or more of a head tracking system, sensor, voice recognition system, body movement sensor, eye-gaze detection system, inertial movement tracking, and microphone.

8. A system comprising:
an interactive head-mounted device including an optical assembly configured to display virtual content and to enable viewing of at least a portion of a surrounding environment, an integrated processor for processing the virtual content for display, and an integrated image source for introducing the virtual content to the optical assembly;
a communications facility configured to connect the interactive head-mounted device to an external device; and
a user action capture device configured to detect a user action as input;
wherein the interactive head-mounted device is configured to enable a command and control scheme for command and control of an external application resident on the external device when a target that indicates that instructions are available to the user is detected by a sensor of the interactive head-mounted device independently of user input, and
wherein the command and control scheme is configured to use user actions captured by the user action capture device as input to the external payment application, to send a request for the instructions to the external application in response to the user actions captured by the user action capture device, and to present the instructions to a user of the interactive head-mounted device responsive to the external application sending the instructions.

9. The system of claim 8 wherein the sensor is a camera.

10. The system of claim 8 wherein the wherein the sensor is a position sensor configured to output coordinates of the interactive head-mounted device.

11. The system of claim 8 , wherein the user action capture device comprises one or more of a head tracking device, a camera, a voice recognition device, a gaze detection device, an inertial movement sensor, and a keyboard.

12. The system of claim 8 , wherein the user actions comprise one or more of user head movements, user eye movements, user voice commands, and user finger movements.

13. The system of claim 8 , wherein the interactive head-mounted device is further configured to enable a second command and control scheme for command and control of a second external application resident on the external device when an event or conditions is detected by the interactive head-mounted device, and wherein the event or condition detected by the interactive head-mounted device comprises an email reception detected by an email application of the interactive head-mounted device.

14. The system of claim 13 , wherein responsive to the email reception being detected, the second command and control scheme is configured to display information from an external payment application on the optical assembly of the interactive head-mounted device.

15. The system of claim 14 , wherein the information from the external payment application displayed on the optical assembly comprises a navigable list of bills to pay, and wherein the input to the external payment application comprises user actions indicating a selection of a bill to pay from the navigable list of bills.

16. The system of claim 8 , wherein the external device comprises one or more of a computer, a payment system, an ATM, and POS system.

17. On a head-mounted display device including an optical assembly through which a surrounding environment and displayed content are viewable, an integrated processor for handling content for display to a user, an integrated image source for introducing the content to the optical assembly, a communications facility configured to connect an external device to the interactive head-mounted display device, a sensor configured to detect an event or condition, and a user action capture device configured to detect a user action as input, a method comprising:
detecting, via data from the sensor and independently of user input, a target in the surrounding environment that indicates that instructions are available to the user;
connecting to the external device via the communications facility;
in response to detecting the target, activating a command and control scheme for command and control of an external application resident on the external device;
detecting a user action via the user action capture device;
translating the user action into input to the external application;
sending a request to the external application for the instructions;
receiving instructions from the external application in response to the input; and
presenting the instructions on the optical assembly.

18. The method of claim 17 , wherein detecting, via data from the sensor, the target comprises detecting, via data from a sensor, the target.

19. The method of claim 17 , wherein detecting the user action via the user action capture device comprises detecting the user action via one or more of a head tracking device, a camera, a voice recognition device, a gaze detection device, an inertial movement sensor, and a keyboard.

20. The method of claim 17 , wherein detecting the user action via the user action capture device comprises detecting one or more of a user head movement, a user eye movement, a user voice command, and a user finger movement.

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Osterhout Ralph F
• Haddick John D
• Lohse Robert Michael
• Cella Charles
• Nortrup Robert J
• Nortrup Edward H

Classifications

• G06F3/017

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title

• G02B2027/014

  comprising information/image processing systems · CPC title

• G02B27/017 Primary

  Head mounted · CPC title

• G06F1/163

  Wearable computers, e.g. on a belt · CPC title

• G02B27/0093

  with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking · CPC title