Gesture library for natural user input

Patent metadata

Field	Value
Publication number	US-9971491-B2
Application number	US-201414151744-A
Country	US
Kind code	B2
Filing date	Jan 9, 2014
Priority date	Jan 9, 2014
Publication date	May 15, 2018
Grant date	May 15, 2018

Abstract

A method to decode natural user input from a human subject. The method includes detection of a gesture and concurrent grip state of the subject. If the grip state is closed during the gesture, then a user-interface (UI) canvas of the computer system is transformed based on the gesture. If the grip state is open during the gesture, then a UI object arranged on the UI canvas is activated based on the gesture.

First claim

The invention claimed is:

1. Enacted in a computing machine operatively coupled to a display and to a vision system having at least one camera, a method to decode natural user input (NUI) from a human subject, the method comprising:
displaying an operating system (OS) shell of an OS of the computing machine, the OS shell including a user-interface (UI) element for each of a plurality of applications of the computing machine,
detecting in three dimensions a hand gesture and concurrent grip state of the human subject while displaying the OS shell;
modifying an extent of visibility of the OS shell based on the gesture if the grip state is close-fisted, wherein modifying the extent of visibility includes displaying a larger portion of the OS shell than currently displayed, to enable different applications to be launched; and
launching an application of the computing machine if the gesture is directed to a UI element presented on the OS shell and associated with the application and if the grip state is open-handed.

2. The method of claim 1 , wherein the gesture includes motion of at least one hand of the subject.

3. The method of claim 1 , further comprising redrawing the OS shell when the extent of visibility of the OS shell is modified.

4. The method of claim 1 , wherein the open-handed grip-state gesture is a push gesture.

5. The method of claim 1 , wherein the UI element is one or more of a tile, icon, or control.

6. The method of claim 1 , wherein detecting the grip state includes resolving a change in the grip state, wherein a closing grip-state change triggers transition to a mode of modifying the extent of visibility of the OS shell, and wherein an opening grip-state change triggers transition to a mode of launching the application.

7. The method of claim 1 wherein displaying the larger portion of the OS shell exposes a plurality of different UI elements, which correspond to the different applications to be launched.

8. The method of claim 1 wherein displaying the larger portion of the OS shell exposes a greater number of UI elements than were exposed prior to the close-fisted gesture.

9. A computing machine configured to decode natural user input (NUI) from a human subject, the computing machine operatively coupled to a display and to a vision system having at least one camera and configured to detect a hand gesture of the subject in three dimensions, the computing machine comprising:
an operating system (OS) accessible to a user of the computing machine via an OS shell, the OS shell being configured to present a user-interface (UI) element for each of a plurality of applications of the computing machine, the OS having an application programming interface (API) to provide, based on the detected hand gesture, input to one or more processes of the computing machine, the input including:
input to signal engagement of the subject as a user of the computing machine,
input to signal disengagement of the subject as a user of the computing machine,
input to shift subsequent input focus to the OS shell upon detection of a close-fisted gesture, wherein shifting subsequent input focus includes reducing a process currently executing on the computing machine to an icon and displaying a larger portion of the OS shell than currently displayed, and
input to launch an application of the computing machine upon detection of an open-handed push gesture directed to a UI element presented on the OS shell and associated with the application.

10. The computing machine of claim 9 , wherein the open-handed gesture includes a press motion followed by a release motion.

11. The computing machine of claim 9 , wherein the input to modify an extent of visibility of the OS is provided upon detection in the vision system of a two-handed stretch or compression gesture with both fists closed.

12. The computing machine of claim 9 , wherein the input to signal engagement of the human subject as a user of the computing machine is provided upon detection of a raised hand and open palm of the subject as the subject faces the display.

13. The computing machine of claim 9 , wherein the input to signal disengagement of the human subject as a user of the computing machine is provided upon detection of both hands of the subject dropping below a pre-defined zone of interaction.

14. The computing machine of claim 9 , wherein the input to the one or more processes further includes input to return to the OS shell from a currently running application, and wherein such input is provided upon detection of a two-handed zoom-out gesture by the subject.

15. The computing machine of claim 9 , wherein the input to the one or more processes further includes input to open or expand the UI element, and wherein such input is provided upon detection of a two-handed zoom-in gesture by the subject.

16. Enacted in a computer system operatively coupled to a vision system, a method to decode natural user input (NUI) from a human subject, the method comprising:
displaying an operating system (OS) shell of an OS of the computing machine, the OS shell including a user-interface (UI) element for each of a plurality of applications of the computing machine;
from a set of gestural inputs detectable by the vision system, selecting a context-appropriate subset based on an operating condition of the computer system;
from a set of actions executable in the computer system, selecting a context-appropriate subset based on the operating condition of the computer system, the operating condition including a grip state of the human subject;
detecting a gestural input from the context-appropriate subset of gestural inputs while displaying the OS shell; and
executing an action from the context-appropriate subset of executable actions responsive to the detected gestural input, the action including modifying an extent of visibility of the OS shell based on the gesture if the grip state is close-fisted, wherein modifying the extent of visibility includes displaying a larger portion of the OS shell than currently displayed, to enable different applications to be launched; and
launching an application of the computing machine if the gesture is directed to a UI element presented on the OS shell and associated with the application and if the grip state is open-handed.

17. The method of claim 16 , wherein the action includes seizing display or input focus from the process currently running.

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Schwesinger Mark
• Yang Emily
• Kapur Jay
• Paolantonio Sergio
• Klein Christian
• Murillo Oscar

Classifications

• G06F3/0304

  Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042) · CPC title

• G06F3/04842

  Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence) · CPC title

• G06F3/013

  Eye tracking input arrangements (G06F3/015 takes precedence) · CPC title

• G06K9/00355

  Physics · mapped topic

• G06F3/017

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title