Semaphore gesture for human-machine interface

Patent metadata

Field	Value
Publication number	US-9791932-B2
Application number	US-201213405365-A
Country	US
Kind code	B2
Filing date	Feb 27, 2012
Priority date	Feb 27, 2012
Publication date	Oct 17, 2017
Grant date	Oct 17, 2017

Abstract

An embodiment of the invention provides a gesture recognition system (GRS) that displays a graphical user interface in which symbols are grouped into “pods” on a display screen, and a user enters a desired symbol into a computer by selecting a pod containing the desired symbol, and then selecting the symbol from a display of the symbols in the selected pod using hand motions without having to contact the display screen or use a manual controller.

First claim

The invention claimed is:

1. A gesture recognition system (GRS) for interfacing a user with a computer, the gesture recognition system comprising:
an imaging system that acquires images of a user;
a display screen; and
a GRS controller configured to generate displays on the display screen based on images of the user acquired by the imaging system, the GRS controller configured to:
generate a first display comprising a plurality of symbol pods, each symbol pod comprising a plurality of symbols;
process images of the user acquired by the imaging system to determine a first spatial location of at least one hand of the user that identifies a particular symbol pod of the plurality of symbol pods;
while displaying the plurality of symbol pods on the display screen, generate a second display based on the determined first spatial location comprising a plurality of tokens and dedicated deposit slots, located outside of and non-overlapping with the particular symbol pod, wherein a different token represents each symbol in the particular symbol pod, and is assigned to a different dedicated deposit slot, wherein a number of the plurality of tokens is equal to a number of the dedicated deposit slots, and is also equal to a number of symbols in the particular symbol pod;
process images of the user acquired by the imaging system to determine a second spatial location of at least one hand of the user that identifies from among the tokens comprised in the second display a token of a desired symbol for entering into the computer;
process images acquired by the imaging system to recognize a motion of the at least one hand that simulates depositing the token comprising the desired symbol identified by the second spatial location into the dedicated deposit slot; and
enter the desired symbol into the computer if such a motion is recognized.

2. The gesture recognition system according to claim 1 wherein each of the symbol pods comprises three symbols.

3. The gesture recognition system according to claim 1 wherein a symbol pod of the plurality of symbol pods comprises a shift button and if the shift button is selected, the GRS controller replaces symbols in at least one of the symbol pods with different symbols.

4. The gesture recognition system according to claim 1 wherein the first and/or second display comprises a text banner for displaying symbols entered into the computer.

5. The gesture recognition system according to claim 1 wherein the GRS controller processes images acquired by the imaging systems to recognize a motion of the at least one hand that simulates shrinking or expanding a portion of the first and/or second display, and respectively shrinks or expands the portion if the motion is recognized.

6. The gesture recognition system according to claim 1 wherein the GRS controller processes images acquire by the imaging systems to recognize a motion of the at least one hand that simulates moving a portion of the first and/or second display and moves the portion if the motion is recognized.

7. The gesture recognition system according to claim 1 wherein the gesture recognition system is configured to receive control signals from a manual controller and to identify a symbol pod comprising a desired symbol to be entered into the computer and to select the desired symbol responsive to the control signals.

8. The gesture recognition system of claim 1 , wherein each token is visually associated with its dedicated deposit slot by an indicium.

9. A method of interfacing a user with a computer, the method comprising:
providing a display comprising a plurality of symbol pods, each symbol pod comprising a plurality of symbols on a display screen;
determining spatial locations for at least one hand of the user from images of the user;
identifying a particular symbol pod of the plurality of symbol pods responsive to determining a first spatial location of the at least one hand;
while displaying the plurality of symbol pods on the display screen, generating a second display based on the determined first spatial location comprising a plurality of tokens and dedicated deposit slots, located outside of and non-overlapping with the particular symbol pod, wherein a different token represents each symbol in the particular symbol pod, and is assigned to a different dedicated deposit slot, wherein a number of the plurality of tokens is equal to a number of the dedicated deposit slots and is also equal to a number of symbols in the particular symbol pod;
determining a second spatial location of the at least one hand that identifies from among the displayed tokens a token of a desired symbol for entering into the computer;
determining, based on images of the user, if the at least one hand simulates depositing the token comprising the desired symbol into the token's dedicated deposit slot and if so, entering the symbol into the computer.

10. The method of claim 9 , wherein each token is visually associated with its dedicated deposit slot by an indicium.

11. A graphical user interface for interfacing a user with a computer, the graphical user interface comprising:
a display screen; and
a graphics display for display on the display screen, the graphics display comprising:
a plurality of user selectable symbol pods, each symbol pod comprising a plurality of symbols for entering into a computer; and
a plurality of tokens and dedicated deposit slots, located outside of and non-overlapping with a user selected symbol pod, wherein:
a token of the plurality of tokens represents each symbol in the user selected symbol pod and is assigned to a different dedicated deposit slot;
a number of the plurality of tokens is equal to a number of the dedicated deposit slots and is also equal to a number of symbols in the particular symbol pod;
wherein a symbol pod of the plurality of symbol pods is selected by acquiring images of a motion of at least one hand of the user that identifies the selected symbol pod, and a token is deposited by a user into its dedicated deposit slot by acquiring images of a motion of at least one hand of the user that simulates depositing the token into its dedicated deposit slot, and the symbol in the token is entered into the computer if such a motion is recognized.

12. The graphical user interface according to claim 11 controllable by user hand motions that do not involve contact with a screen or use of a manual controller.

13. The graphical user interface of claim 11 , wherein each token is visually associated with its dedicated deposit slot by an indicium.

Assignees

• Yahav Giora
• Goren Uri
• Bahat Yuval
• Mendels Omri
• Specktor Bella
• Microsoft Technology Licensing Llc

Inventors

• Yahav Giora
• Goren Uri
• Bahat Yuval
• Mendels Omri
• Specktor Bella

Classifications

• G06F3/017 Primary

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title

• G06F3/0236

  using selection techniques to select from displayed items · CPC title

• G06F3/04842

  Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence) · CPC title