Multimodal input system

Patent metadata

Field	Value
Publication number	US-2016350071-A1
Application number	US-201615151157-A
Country	US
Kind code	A1
Filing date	May 10, 2016
Priority date	Nov 1, 2010
Publication date	Dec 1, 2016
Grant date	—

Abstract

The subject disclosure relates to user input into a computer system, and a technology by which one or more users interact with a computer system via a combination of input modalities. When the input data of two or more input modalities are related, they are combined to interpret an intended meaning of the input. For example, speech when combined with one input gesture has one intended meaning, e.g., convert the speech to verbatim text for consumption by a program, while the exact speech when combined with a different input gesture has a different meaning, e.g., convert the speech to a command that controls the operation of that same program.

First claim

What is claimed is:

1 . In a computing environment, a method performed at least in part on at least one processor, the method comprising:
receiving sets of input data corresponding to a plurality of input modalities, the received sets of input data including a first set of input data and a second set of input data, the first set of input data being associated with a first input modality from the plurality of input modalities, the second set of input data being associated with a second input modality from the plurality of input modalities; selecting the first set of input data and the second set of input data; accessing a dictionary to determine a meaning of the first and second set of input data; and providing output data for input by the program, the output data corresponding to the meaning of the first and second set of input data.

2 . The method of claim 1 , further comprising:
receiving the first set of input data from a first input device; and receiving the second set of input data from a second input device.

3 . The method of claim 2 wherein the first input device is associated with a first user and the second input device is associated with a second user.

4 . The method of claim 1 wherein accessing a dictionary to determine a meaning of the first and second set of input data comprises accessing a dictionary to determine a combined meaning of the first and second set of input data.

5 . The method claim 1 wherein at least one of the first set of input data and the second set of input data is received as input that is synthesized from output of another device.

6 . The method of claim 1 further comprising determining that the first set of input data is related to the second set of input data by accessing preference data.

7 . The method of claim 6 wherein determining that the first set of input data is related to the second set of input data further comprises accessing data provided by at least one sensor.

8 . A system, comprising:
a plurality of input devices that provide raw data corresponding to one or more input modalities; an input processor configured to process the raw data into input events; one or more sensors that provide environmental input; and a recommendation engine configured to:
select a subset of the input events based on one or more of the following: user defined preferences, and a current operating mode of a program;
determine a meaning of the subset of input events; and
provide output data for input by the program, the output data corresponding to the meaning of the subset of input events and the environmental input provided by the one or more sensors.

9 . The system of claim 8 wherein a first input device from the plurality of input devices inputs raw data from one user and a second input device from the plurality of input devices inputs raw data from another user.

10 . The system of claim 8 wherein at least one of the plurality of input devices is coupled to the system via a remote connection.

11 . The system of claim 8 wherein the plurality of input devices include one or more of the following: a camera, a microphone, a digitizer, a pen, and a pointing device.

12 . The system of claim 8 wherein the input processor includes one or more of the following: an audio processing component, a visual processing component, a touch processing component, a gesture processing component, a pointer processing component and a text processing component.

13 . The system of claim 8 further comprising a personalization engine configured to communicate with the recommendation engine to provide personalization information that the recommendation engine uses in determining the meaning of the subset of input events.

14 . The system of claim 13 wherein the personalization engine accesses multiple dictionaries to provide the personalization information.

15 . The system of claim 8 further comprising a combined dictionary corresponding to two or more input modalities, wherein the combined dictionary is selected from a plurality of combined dictionaries based on a current operating mode of a program configured to receive the meaning of the subset of input events.

16 . The system of claim 8 wherein the recommendation engine is further configured to use environmental input in determining the meaning of the subset of input events, wherein the subset of input events corresponding to at least one of the plurality of input devices is restricted.

17 . The system of claim 8 wherein the recommendation engine is further configured to use preference data in determining the meaning of the subset of input events.

18 . One or more computer-readable storage media having computer-executable instructions, that cause a processor to perform operations comprising:
receiving sets of input data corresponding to a plurality of input modalities, the received sets of input data including a first set of input data and a second set of input data, the first set of input data being associated with a first input modality from the plurality of input modalities, the second set of input data being associated with a second input modality from the plurality of input modalities; selecting the first set of input data and the second set of input data; accessing a dictionary to determine a meaning of the first and second set of input data; and providing output data for input by the program, the output data corresponding to the meaning of the first and second set of input data.

19 . The one or more computer-readable storage media of claim 18 wherein an intent of the user is based at least in part on the first input modality and the second input modality.

20 . The one or more computer-readable storage media of claim 18 wherein the dictionary is associated with the first input modality and the second input modality.

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Murillo Oscar E
• Galore Janet E
• Cluts Jonathan C
• Estrada Colleen G
• Koenig Michael
• Creasey Jack
• Bhattacharyay Subha

Classifications

• G06F2203/0381

  Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer · CPC title

• G06F9/451

  Execution arrangements for user interfaces · CPC title

• G06F3/167 Primary

  Audio in a user interface, e.g. using voice commands for navigating, audio feedback · CPC title

• G06F3/017 Primary

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title

• G06F2203/0382

  Plural input, i.e. interface arrangements in which a plurality of input device of the same type are in communication with a PC · CPC title