Manipulating an object utilizing a pointing device

Patent metadata

Field	Value
Publication number	US-10488950-B2
Application number	US-10436008-A
Country	US
Kind code	B2
Filing date	Apr 16, 2008
Priority date	Feb 7, 2002
Publication date	Nov 26, 2019
Grant date	Nov 26, 2019

Abstract

The present invention is directed toward a system and process that controls a group of networked electronic components using a multimodal integration scheme in which inputs from a speech recognition subsystem, gesture recognition subsystem employing a wireless pointing device and pointing analysis subsystem also employing the pointing device, are combined to determine what component a user wants to control and what control action is desired. In this multimodal integration scheme, the desired action concerning an electronic component is decomposed into a command and a referent pair. The referent can be identified using the pointing device to identify the component by pointing at the component or an object associated with it, by using speech recognition, or both. The command may be specified by pressing a button on the pointing device, by a gesture performed with the pointing device, by a speech recognition event, or by any combination of these inputs.

First claim

The invention claimed is:

1. A computer-readable storage device having stored thereon computer-executable instructions comprising:
identifying a position and an orientation of a pointing device in three-dimensional space;
determining that the pointing device is directed to an object based on the position and the orientation of the pointing device in three-dimensional space;
determining that an input sequence of sensor values output by the pointing device matches a matching prototype sequence from a set of stored prototype sequences, wherein each stored prototype sequence represents a sequence of said sensor values that are generated if a user performs a unique gesture representing a different control action using the pointing device, wherein the matching prototype sequence is determined at least by comparing the matching prototype sequence to the input sequence of sensor values and by comparing the input sequence of sensor values to one or more versions of the matching prototype sequence that are scaled up and down in amplitude and warped in time; and
controlling a computer system based on a command and referent pair, wherein the referent is determined based on the position and the orientation of the pointing device in three-dimensional space, and wherein the command is determined based on a gesture associated with the matching prototype sequence.

2. The computer-readable storage device of claim 1 , wherein the sensor values are provided by an output of an accelerometer.

3. The computer-readable storage device of claim 1 , wherein identifying the position further comprises:
receiving data associated with one or more infrared lights detected by a camera.

4. The computer-readable storage device of claim 3 , wherein the data associated with one or more infrared lights detected by the camera describes one or more images that are analyzed to identify the position.

5. The computer-readable storage device of claim 1 , wherein the input sequence of sensor values includes sensor values that are provided by an output of a gyroscope.

6. A method for multimodal electronic component control comprising:
identifying a position and an orientation of a pointing device in three-dimensional space;
determining that the pointing device is directed to an object based on the position and the orientation of the pointing device in three-dimensional space;
determining that an input sequence of sensor values output by the pointing device matches a matching prototype sequence from a set of stored prototype sequences, wherein each stored prototype sequence represents a sequence of said sensor values that are generated if a user performs a unique gesture representing a different control action using the pointing device, wherein the matching prototype sequence is determined at least by comparing the matching prototype sequence to the input sequence of sensor values and by comparing the input sequence of sensor values to one or more versions of the matching prototype sequence that are scaled up and down in amplitude and warped in time; and
controlling a computer system based a command and referent pair, wherein the referent is determined based on the position and the orientation of the pointing device in three-dimensional space, and wherein the command is determined based on a gesture associated with the matching prototype sequence.

7. The method of claim 6 , wherein identifying the position further comprises:
receiving data associated with one or more infrared lights detected by a camera.

8. The method of claim 7 , wherein the data associated with one or more infrared lights detected by the camera describes one or more images that are analyzed to identify the position.

9. The method of claim 6 , wherein the sensor values are provided by an output of an accelerometer.

10. The method of claim 6 , further comprising:
computing a similarity between the input sequence of sensor values output by the pointing device and each stored prototype sequence;
determining if the computed similarity between the input sequence and any stored prototype sequence exceeds a prescribed similarity threshold; and
whenever it is determined that one of the computed similarities exceeds the similarity threshold, determining that the stored prototype sequence associated with the computed similarity matches the input sequence.

11. A system for multimodal electronic component control comprising:
one or more cameras that capture one or more images of a pointing device in a three-dimensional space;
a processing component configured to perform instructions comprising:
receiving data associated with the one or more images;
identifying a position and an orientation of the pointing device in the three-dimensional space based on the received data;
determining that the pointing device is directed to an object based on the position and the orientation of the pointing device in three-dimensional space;
determining that an input sequence of sensor values output by the pointing device matches a matching prototype sequence from a set of stored prototype sequences, wherein each stored prototype sequence represents a sequence of said sensor values that are generated if a user performs a unique gesture representing a different control action using the pointing device, wherein the matching prototype sequence is determined at least by comparing the matching prototype sequence to the input sequence of sensor values and by comparing the input sequence of sensor values to one or more versions of the matching prototype sequence that are scaled up and down in amplitude and warped in time; and
controlling the system based on a command and referent pair, wherein the referent is determined based on the position and the orientation of the pointing device in three-dimensional space, and wherein the command is determined based on a gesture associated with the matching prototype sequence.

12. The system of claim 11 , wherein the sensor values are provided by an output of an accelerometer.

13. The system of claim 11 , wherein the data associated with the one or more images comprises data associated with one or more infrared lights detected by the camera.

14. The system of claim 11 , wherein the instructions further comprise:
computing a similarity between the input sequence of sensor values output by the pointing device and each stored prototype sequence;
determining if the computed similarity between the input sequence and any stored prototype sequence exceeds a prescribed similarity threshold; and
whenever it is determined that one of the computed similarities exceeds the similarity threshold, determining that the stored prototype sequence associated with the computed similarity matches the input sequence.

Assignees

• Wilson Andrew
• Microsoft Technology Licensing Llc

Inventors

• Wilson Andrew

Classifications

• G06F3/038

  Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry · CPC title

• G06F3/0346 Primary

  with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors · CPC title

• G08C17/00

  Arrangements for transmitting signals characterised by the use of a wireless electrical link · CPC title

• G08C2201/31

  Voice input · CPC title

• G08C2201/41

  Remote control of gateways · CPC title