Visual gestures for a head mounted device

Patent metadata

Field	Value
Publication number	US-9652047-B2
Application number	US-201615052775-A
Country	US
Kind code	B2
Filing date	Feb 24, 2016
Priority date	Feb 25, 2015
Publication date	May 16, 2017
Grant date	May 16, 2017

Abstract

Visual gestures in a display device allow a user to select and activate features in a display of the display device. A sensor of the display device tracks an eye gaze of a user directed at a display of the display device. A visual gesture module identifies a predefined trigger zone in the display. A virtual object application displays a virtual object in the display based on the eye gaze of the user and the predefined trigger zone.

First claim

What is claimed is:

1. A head mounted display device comprising:
a transparent display operatively positioned to present virtual content to a user wearing the head mounted display device while allowing the user to view physical objects through the transparent display;
one or more eye gaze sensors configured to track at least a first pupil of the user to determine an eye gaze of the user, wherein the eye gaze of the user indicates a location on the transparent display in a line of sign of the user;
one or more computer processors; and
a memory storing instructions that, when executed by the one or computer processors, cause the head mounted display device to:
monitor the eye gaze of the user wearing the head mounted display device;
determine, based on monitoring the eye gaze of the user, that a duration of time that the line of sight of the user is within a predefined trigger zone of the transparent display exceeds a threshold duration for the predefined trigger zone, wherein the predefined trigger zone corresponds to a first physical object visible to the user through the predefined trigger zone of the head mounted display; and
in response to determining that the duration of time that the line of sight of the user is within the predefined trigger zone of the transparent display exceeds the threshold duration, cause a display of a virtual object in the predefined trigger zone of the transparent display.

2. The head mounted display device of claim 1 , wherein the one or more eye gaze sensors are configured to identify a direction and a duration of the eye gaze of the user.

3. The head mounted display device of claim 2 , wherein the instructions further cause the head mounted display device to change a state of the virtual object based on a change in the direction and duration of the eye gaze of the user.

4. The head mounted display device of claim 2 , wherein the predefined trigger zone includes a plurality of gradient zones, each gradient zones associated with a corresponding predefined trigger duration.

5. The head mounted display device of claim 2 , wherein the instructions further cause the head mounted display device to:
identify a spatial and temporal pattern of the eye gaze of the user;
identify a predefined spatial and temporal pattern; and
display the virtual object in the display based on the spatial and temporal pattern of the eye gaze of the user matching the predefined spatial and temporal pattern.

6. The head mounted display device of claim 2 , wherein the instructions further cause the head mounted display device to:
identify a focal depth of the eye gaze of the user;
identify a predefined focal depth range; and
display the virtual object in the transparent display based on the focal depth of the eye gaze of the user being with the predefined focal depth range.

7. The head mounted display device of claim 2 , further comprising:
a voltage sensor connected to a skin of the user, the voltage sensor configured to identify a physical and mental state of the user, wherein the instructions further cause the head mounted display device to:
identify a predefined physical and mental state corresponding to a second virtual object; and
display the second virtual object in the transparent display based on a combination of the physical and mental state of the user corresponding to the predefined physical and mental state and the eye gaze of the user intersecting the predefined trigger zone.

8. The head mounted display device of claim 2 , wherein the instructions further cause the head mounted display device to:
identify that the eye gaze of the user is within a predefined distance threshold of the predefined trigger zone; and
display a visual indicator being pulled towards the predefined trigger zone.

9. The head mounted display device of claim 2 , wherein the instructions further cause the head mounted display device to:
identify a predefined trigger point associated with a body part of the user; and
display a second virtual object in the transparent display based on a first eye gaze of the user directed at the body part of the user followed by a second eye gaze of the user directed at the predefined trigger zone associated with the body part.

10. A computer-implemented method comprising:
monitoring, by a head mounted display device, an eye gaze of a user wearing the head mounted display device, the head mounted display device including a transparent display operatively positioned to present virtual content to the user while allowing the user to view physical objects through the transparent display, the head mounted display including one or more eye gaze sensors configured to track at least a first pupil of the user to determine the eye gaze of the user, wherein the eye gaze of the user indicates a location on the transparent display in a line of sight of the user;
determining, based on monitoring the eye gaze of the user, that a duration of time that the line of sight of the user is within a predefined trigger zone of the transparent display exceeds a threshold duration for the predefined trigger zone, wherein the predefined trigger zone corresponds to a first physical object visible to the user through the predefined trigger zone of the head mounted display; and
in response to determining that the duration of time that the line of sight of the user is within the predefined trigger zone of the transparent display exceeds the threshold duration, causing a display of a virtual object in the predefined trigger zone of the transparent display.

11. The computer-implemented method of claim 10 , further comprising:
identifying a direction and a duration of the eye gaze of the user.

12. The computer-implemented method of claim 11 , further comprising:
changing a state of the virtual object based on a change in the direction and duration of the eye gaze of the user.

13. The computer-implemented method of claim 11 , wherein the predefined trigger zone includes a plurality of gradient zones, each gradient zones associated with a corresponding predefined trigger duration.

14. The computer-implemented method of claim 11 , further comprising:
identifying a spatial and temporal pattern of the eye gaze of the user;
identifying a predefined spatial and temporal pattern; and
displaying the virtual object in the display based on the spatial and temporal pattern of the eye gaze of the user matching the predefined spatial and temporal pattern.

15. The computer-implemented method of claim 11 , further comprising:
identifying a focal depth of the eye gaze of the user;
identifying a predefined focal depth range; and
displaying the virtual object in the transparent display based on the focal depth of the eye gaze of the user being with the predefined focal depth range.

16. The computer-implemented method of claim 11 , further comprising:
identifying a physical and mental state of the user using a voltage sensor connected to a skin of the user;
identifying a predefined physical and mental state corresponding to a second virtual object; and
displaying the second virtual object in the transparent display based on a combination of the physical and mental state of the user corresponding to the predefined physical and mental state and the eye gaze of the user intersecting the predefined trigger zone.

17. The computer-implemented method of claim 11 , further comprising:
identifying a predefined trigger point associated with a body part of the user; and
displaying a second virtual object in the transparent display based on a first eye gaze of the user dir

Assignees

• Daqri Llc

Inventors

• Mullins Brian
• Kammerait Matthew
• Broaddus Christopher
• Crispin Sterling
• Kazansky Lucas

Classifications

• G02B2027/014

  comprising information/image processing systems · CPC title

• G06F3/017 Primary

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title

• G02B2027/0187

  slaved to motion of at least a part of the body of the user, e.g. head, eye · CPC title

• G06F3/0481

  based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance · CPC title

• G06F3/013

  Eye tracking input arrangements (G06F3/015 takes precedence) · CPC title