Command input by hand gestures captured from camera

Patent metadata

Field	Value
Publication number	US-9317124-B2
Application number	US-53637306-A
Country	US
Kind code	B2
Filing date	Sep 28, 2006
Priority date	Sep 28, 2006
Publication date	Apr 19, 2016
Grant date	Apr 19, 2016

Abstract

A method and system for invoking an operation of a communication terminal in response to registering and interpreting a predetermined motion or pattern of an object. An input is received, the image data of the object is captured and the object in the image data is identified.

First claim

The invention claimed is:

1. A method for invoking an operation of a communication terminal in response to registering and interpreting a predetermined motion or pattern of an object, the method comprising:
receiving an indication of a predetermined motion or pattern of a single occurrence of a hand input gesture received from a digital camera of the communication terminal;
identifying at least a first and second centroid in the object;
identifying at least one convex hull in the object;
matching the hand input gesture received from the digital camera with a reference gesture object selected from a set of reference gesture objects based on a positional relationship of the at least first and second centroids in the object and a change in size of the convex hull; and
executing a command related to an item of a display of the communication terminal in response to and upon matching the single occurrence of the hand input gesture received from the digital camera with the reference gesture object.

2. The method according to claim 1 , wherein said interpretation of a motion or pattern is performed by a software of said terminal.

3. The method according to claim 1 , wherein said operation involves provision of a command input to the communication terminal based on the single occurrence of the hand input gesture, and that said method comprises:
capturing image data of said hand input gesture using the digital camera of the communication terminal;
identifying an object in said image data;
recognizing the object as a hand;
recognizing and associating characteristics of said object as a gesture of said hand with a single, first reference gesture from a set of predetermined reference gestures;
providing a command input associated with said single reference gesture.

4. The method according to claim 3 , wherein said identification involves classifying skin color.

5. The method according to claim 4 , wherein said skin color classification comprises performing Gaussian mixture modeling.

6. The method according to claim 4 , wherein said skin color classification involves color space analysis and/or probability analysis.

7. The method according to claim 6 , wherein said color space analysis involves converting image data to chrominance plane (CbCr) color space image data.

8. The method according to claim 3 , wherein said object recognition involves eliminating visual noise using connected component extraction.

9. The method according to claim 8 , wherein said connected component extraction comprises any of the following:
determining aspect ratio of the object;
determining size of object compared to image size;
determining regions connecting to the borders of the input image; and
wherein said noise is eliminated in case the following requirements are fulfilled:
said aspect ratio is within 10;
said object size is greater than a predetermined value set in relation to the input image size; and
there is either only one region connecting to the borders of the input image, or a plurality of regions which do not meet the other requirements.

10. The method according to claim 3 further comprising determining orientation of said hand and involving:
determining a Karhunen-Loeve (KL) axis orientation of said object;
determining a first geometric centerpoint of said object; and
determining a second geometric centerpoint of a convex polygon of said object, and
wherein the orientation of said KL axis is used to determine a positional relationship of said first and second centerpoints.

11. The method according to claim 10 , wherein said orientation determining arrives at one of the following:
performing a first operation if said KL-axis extends along a first direction, and the first and second centerpoints are displaced in a first displacement direction in relation to each other, essentially along said first direction;
performing a second operation if said KL-axis extends along said first direction, and the first and second centerpoints are displaced reversely in said displacement direction in relation to each other, essentially along said first direction;
performing a third operation if said KL-axis extends along a second direction, essentially perpendicular to said first direction, and the first and second centerpoints are displaced in a second displacement direction in relation to each other, essentially along said second direction;
performing a fourth operation if said KL-axis extends along said second direction, and the first and second centerpoints are displaced reversely in said displacement direction in relation each other, essentially along said second direction;
performing a fifth operation if said centerpoints are essentially superpositioned, and a first area of said object of said gesture is less than at least half a second area of a previously determined object of a previously recognized gesture;
performing a sixth operation if said centerpoints are essentially superpositioned, a first area of said object of said gesture is greater than at least twice a second area of a previously determined object of a previously recognized gesture, and said gesture corresponds to said previously recognized gesture;
performing a seventh operation if said centerpoints are essentially superpositioned, a first area of said object of said gesture is greater than at least twice a second area of a previously determined object of a previously recognized gesture, and said gesture does not correspond to said previous gesture.

12. The method according to claim 11 , wherein said first, second, third, and fourth operations correspond to moving focus up, down, left, and right respectively, and said fifth, sixth, and seventh operations correspond to open an item, close an item, and stop the focus motion respectively.

13. The method according to claim 1 , wherein said registering is performed using a camera of said communication terminal.

14. The method according to claim 1 , further comprising a step of activation by proximity detection.

15. A computer-readable storage medium having computer-executable components embodied there, said computer-readable medium being configured to execute the method according to claim 1 .

16. The computer-readable storage medium according to claim 15 , further being configured to:
receive an input;
capture image data of said object;
identify said object in said image data;
recognize said object as a hand;
recognize and associate characteristics of said object as a gesture of said hand with a first reference gesture from a set of predetermined reference gestures;
provide a command input associated with said reference object.

17. The method of claim 1 wherein the item is a document, a folder, a web page, a call, a message, a contact item, a multimedia content item, a reminder, or a textual entry.

18. The method of claim 17 wherein the hand gesture corresponds to a command to maneuver a menu on the display or toggle between items on the display.

19. An apparatus comprising:
a processor; and
a memory including computer program code, wherein the memory and the computer program code are configured to, along with the processor, cause the apparatus at least to perform:
invoking an operation of the apparatus in response to registering and interpreting a predetermined motion or pattern of an object;
receiving an indication of a predetermined motion or pattern of a single occurrence of a hand input gesture received from an imaging unit;
identifying at least a first and second centroid in the object;
identifying

Assignees

• Kongqiao Wang
• Takala Roope
• Fang Yikai
• Lu Hanqing
• Liu Qingshan
• Nokia Technologies Oy

Inventors

• Kongqiao Wang
• Takala Roope
• Fang Yikai
• Lu Hanqing
• Liu Qingshan

Classifications

• G06K9/00375

  Physics · mapped topic

• G06F3/017 Primary

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title

• G06V40/107

  Static hand or arm · CPC title