Sessionless pointing user interface

Patent metadata

Field	Value
Publication number	US-9218063-B2
Application number	US-201213592352-A
Country	US
Kind code	B2
Filing date	Aug 23, 2012
Priority date	Aug 24, 2011
Publication date	Dec 22, 2015
Grant date	Dec 22, 2015

Abstract

A method, including receiving, by a computer, a sequence of three-dimensional maps containing at least a hand of a user of the computer, and identifying, in the maps, a device coupled to the computer. The maps are analyzed to detect a gesture performed by the user toward the device, and the device is actuated responsively to the gesture.

First claim

We claim:

1. A method, comprising:
identifying, by a computer coupled to a three-dimensional (3D) sensing device and a display, different, respective locations of multiple controllable devices other than the display;
receiving, by the computer from the 3D sensing device, a sequence of three-dimensional maps containing at least a head and a hand of a user of the computer;
detecting in the maps a gaze direction of the user that is directed toward a given device among the multiple controllable devices;
defining a region in space with an apex at the head of the user and a base encompassing the given device;
defining an interaction zone within the region;
defining an angle threshold;
defining a minimum time period;
analyzing the maps to detect a gesture performed by the hand within the defined interaction zone that is directed toward an identified location of the given device among the multiple controllable devices,
wherein the gesture comprises extending an elbow associated with the hand at an angle greater than or equal to the angle threshold, extending the hand toward the device and pausing the hand for the minimum time period; and
actuating the given device responsively to the gesture.

2. The method according to claim 1 , wherein the gesture is selected from a list comprising a pointing gesture, a grab gesture and a release gesture.

3. The method according to claim 1 , and comprising receiving a vocal command from the user, and actuating the device in response to the gesture and the vocal command.

4. The method according to claim 1 , and comprising communicating between the computer and a further device upon detecting, in the maps, a subsequent gesture directed toward the further device.

5. The method according to claim 1 , wherein identifying the respective locations comprises performing, by the computer, an initialization step comprising:
identifying, by the computer, the controllable devices that are in proximity to the 3D sensing device; and
directing the user to point to each of the identified controllable devices.

6. An apparatus, comprising:
a three-dimensional sensing device;
a display; and
a computer configured to identify different, respective locations of multiple controllable devices other than the display, to receive from the three-dimensional sensing device a sequence of three-dimensional maps containing at least a head and a hand of a user of the computer, to detect in the maps a gaze direction of the user that is directed toward a given device among the multiple controllable devices, to define a region in space with an apex at the head of the user and a base encompassing the given device, to define an interaction zone within the region, to analyze the maps to detect a gesture performed by the hand within the defined interaction zone that is directed toward an identified location of the given device among the multiple controllable devices, and to actuate the given device responsively to the gesture,
wherein the computer is configured to define an angle threshold and a minimum time period, and wherein the gesture detected by the computer comprises extending an elbow associated with the hand at an angle greater than or equal to the angle threshold, extending the hand toward the given device, and pausing the hand for the minimum time period.

7. The apparatus according to claim 6 , wherein the computer is configured to select the gesture from a list comprising a pointing gesture, a grab gesture and a release gesture.

8. The apparatus according to claim 6 , wherein the computer is configured to receive a vocal command from the user, and to actuate the given device in response to the gesture and the vocal command.

9. The apparatus according to claim 6 , wherein the computer is configured to communicate with a further device among the multiple controllable devices upon detecting, in the maps, a subsequent gesture directed toward the further device.

10. A computer software product comprising a non-transitory computer-readable medium, in which program instructions are stored, which instructions, when read by a computer executing a non-tactile user interface and coupled to a display, cause the computer to:
identify different, respective locations of multiple controllable devices other than the display,
receive from a three-dimensional sensing device that is coupled to the computer a sequence of three-dimensional maps containing at least a head and a hand of a user of the computer,
detect in the maps a gaze direction of the user that is directed toward a given device among the multiple controllable devices,
define a region in space with an apex at the head of the user and a base encompassing the given device,
define an interaction zone within the region,
analyze the maps to detect a gesture performed by the hand within the defined interaction zone that is directed toward an identified location of the given device among the multiple controllable devices, and
actuate the given device responsively to the gesture,
wherein the instructions cause the computer to define an angle threshold and a minimum time period, and wherein the gesture detected by the computer comprises extending an elbow associated with the hand at an angle greater than or equal to the angle threshold, extending the hand toward the given device, and pausing the hand for the minimum time period.

Assignees

• Galor Micha
• Pokrass Jonathan
• Hoffnung Amir
• Or Ofir
• Apple Inc

Inventors

• Galor Micha
• Pokrass Jonathan
• Hoffnung Amir
• Or Ofir

Classifications

• G06T2207/30196

  Human being; Person · CPC title

• G06F3/013

  Eye tracking input arrangements (G06F3/015 takes precedence) · CPC title

• G06F3/017 Primary

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title

• G06F3/167

  Audio in a user interface, e.g. using voice commands for navigating, audio feedback · CPC title

• G06T7/75

  involving models · CPC title