Multimodal interaction with near-to-eye display

Patent metadata

Field	Value
Publication number	US-10095033-B2
Application number	US-201214412357-A
Country	US
Kind code	B2
Filing date	Jul 27, 2012
Priority date	Jul 27, 2012
Publication date	Oct 9, 2018
Grant date	Oct 9, 2018

Abstract

The invention relates to a method, comprising tracking the gaze of an eye of a user by a first camera, wherein a position of the gaze determines a place for the cursor on the screen of a near-to-eye display (NED), wherein the movement of the gaze is configured to move the cursor on the screen and when the movement of the gaze is detected to be still or moving only slightly; observing a gesture of one hand of the user by a second camera, and executing a further operation if at least one gesture of the hand is detected. The invention further relates to an apparatus and a computer program product configured to perform the method.

First claim

What is claimed is:

1. A method, comprising:
tracking a gaze of an eye of a user by an inside-looking camera of a near-to-eye display, wherein a position of the gaze determines a place for a cursor on the screen of the near-to-eye display, wherein the movement of the gaze is configured to move the cursor on the screen of the near-to-eye display and when the movement of the gaze is detected to be still or moving only slightly,
obtaining a first position of a hand of the user by a side down-looking camera of the near-to-eye display,
obtaining a second position of the hand of the user by the side down-looking camera of the near-to-eye display,
computing relative movement of the hand based on the first and second positions of the hand,
positioning the cursor on the basis of the relative movement of the hand,
in an instance at least one gesture of the hand is detected by the side down-looking camera, activating a manual control of the user interface by adjusting the position of the cursor on the screen of the near-to-eye display;
adjusting the position of the cursor on the screen of the near-to-eye display on a basis of further relative movement of the hand in an instance in which the cursor is outside an area of a menu option or a functional icon displayed on the screen; and
in an instance at least one gesture is no longer detected by the side down-looking camera, resuming a gaze control of the user interface.

2. The method of claim 1 , wherein the further operation depends on the detected gesture.

3. The method of claim 1 , wherein the detected gesture is a movement of a fist, shaking of a palm or movement of a palm from or to the second camera.

4. The method of claim 1 , wherein the further operation further depends on the position of the cursor on the screen of the near-to-eye display.

5. The method of claim 1 , wherein the further operation is:
selecting a menu option or a functional icon displayed on the screen of the near-to-eye display if the cursor is on the menu option or functional icon.

6. The method of claim 1 , wherein the further operation is:
zooming in or out the view of the screen of the near-to-eye display.

7. The method of claim 1 , wherein the further operation is:
returning, removing or deleting.

8. The method of claim 1 , further comprising:
observing blinking of the eye of the user if no gesture of the hand is detected and if the cursor is on the area of a menu option or a functional icon displayed on the screen, and
selecting the menu option or the functional icon if blinking of the eye of the user is detected.

9. An apparatus comprising at least one processor, and at least one memory including a computer program code, said at least one memory and the computer program code being configured, with the processor, to cause the apparatus to perform at least the following:
track a gaze of one eye of a user by an inside-looking camera of a near-to-eye display, wherein a position of the gaze determines a place for a cursor on a screen of the near-to-eye display of the apparatus, wherein a movement of the gaze is configured to move the cursor on the screen of the near-to-eye display and when the movement of the gaze is detected to be still or moving only slightly,
obtain a first position of a hand of the user by a side down-looking camera of the near-to-eye display,
obtain a second position of the hand of the user by the side down-looking camera of the near-to-eye display,
compute relative movement of the hand based on the first and second positions of the hand,
position the cursor on the basis of the relative movement of the hand,
in an instance at least one gesture of the hand is detected by the side down-looking camera, activate a manual control of the user interface by adjusting the position of the cursor on the screen of the near-to-eye display;
adjust the position of the cursor on the screen of the near-to-eye display on a basis of further relative movement of the hand in an instance in which the cursor is outside an area of a menu option or a functional icon displayed on the screen; and
in an instance that the at least one gesture is no longer detected by the side down-looking camera, resume a gaze control of the user interface.

10. The apparatus of claim 9 , wherein the further operation depends on the detected gesture.

11. The apparatus of claim 9 , wherein the detected gesture may be the movement of a fist, the shaking of a palm and the movement of a palm from or to the second camera.

12. The apparatus of claim 9 , wherein the further operation further depends on the position of the cursor on the screen of the near-to-eye display.

13. The apparatus of claim 9 , wherein the further operation is:
select a menu option or a functional icon displayed on the screen of the near-to-eye display if the cursor is on the menu option or functional icon.

14. The apparatus of claim 9 , wherein the further operation is:
zoom in or out the view of the screen of the near-to-eye display.

15. The apparatus of claim 9 , wherein the further operation is:
return, remove or delete.

16. The apparatus of claim 9 , wherein the computer program code is further configured, with the processor, to cause the apparatus to perform:
observe blinking of the eye of the user if no gesture of the hand is detected and if the cursor is on the area of a menu option or a functional icon displayed on the screen, and
select the menu option or functional icon of the cursor on the screen of the near-to-eye display if blinking of the eye of the user is detected.

17. The apparatus of claim 9 , wherein said apparatus is a near-to-eye display device.

18. The apparatus of claim 17 , wherein said near-to-eye display device is a terminal device for a mobile phone.

Assignees

• Wang Kongqiao
• Lantz Vuokko
• Li Jiangwei
• Xu Lei
• Huopaniemi Jyri
• Nokia Technologies Oy

Inventors

• Wang Kongqiao
• Lantz Vuokko
• Li Jiangwei
• Xu Lei
• Huopaniemi Jyri

Classifications

• G06F3/04812

  Interaction techniques based on cursor appearance or behaviour, e.g. being affected by the presence of displayed objects · CPC title

• G02B27/0172 Primary

  characterised by optical features · CPC title

• G06F3/013

  Eye tracking input arrangements (G06F3/015 takes precedence) · CPC title

• G06F3/011 Primary

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• G02B2027/0178

  Eyeglass type (eyeglass details G02C) · CPC title