Virtual keyboard based on adaptive language model

Patent metadata

Field	Value
Publication number	US-11327651-B2
Application number	US-202016789079-A
Country	US
Kind code	B2
Filing date	Feb 12, 2020
Priority date	Feb 12, 2020
Publication date	May 10, 2022
Grant date	May 10, 2022

Abstract

Disclosed herein are related to systems and methods for providing inputs through a virtual keyboard with an adaptive language model. In one approach, one or more processors determine whether a user intended to provide semantically meaningful characters or not, when providing a hand motion or a hand pose with respect to a virtual keyboard. The virtual keyboard may be located on a surface without physical keys. In one approach, the one or more processors determine an input to the virtual keyboard based on the hand motion or the hand pose. In one approach, the one or more processors determine weight of a language model according to the determined user intention. In one approach, the one or more processors modify the detected input according to the determined weight of the language model.

First claim

What is claimed is:

1. A method comprising:
determining, by one or more processors, a level of focus of a user when providing a hand motion with respect to a virtual keyboard, using data from at least one sensor including an image sensor communicably coupled to the one or more processors, the data indicative of a gaze or face direction of the user relative to a position of the virtual keyboard, wherein determining the level of focus of the user when providing the hand motion includes determining, by the one or more processors, a speed of the hand motion using the data acquired by the image sensor;
detecting, by the one or more processors via the image sensor, an input to the virtual keyboard based on the hand motion;
determining, by the one or more processors, weight of a language model according to the data indicative of the gaze or face direction of the user relative to the position of the virtual keyboard; and
modifying, by the one or more processors, the detected input according to the determined weight of the language model.

2. The method of claim 1 , wherein determining the level of focus of the user when providing the hand motion includes:
determining, by the one or more processors, an orientation of a head of the user.

3. The method of claim 2 , wherein determining the weight of the language model includes:
determining, by the one or more processors, the weight to be a first value, in response to determining that the head is oriented to face towards the virtual keyboard, and determining, by the one or more processors, the weight to be a second value, in response to determining that the head is oriented to face away from the virtual keyboard, the second value higher than the first value.

4. The method of claim 1 , wherein determining the level of focus of the user when providing the hand motion includes:
determining, by the one or more processors, the gaze direction of the user.

5. The method of claim 4 , wherein determining the weight of the language model includes:
determining, by the one or more processors, the weight to be a first value, in response to determining that the gaze direction of the user is directed to the virtual keyboard, and
determining, by the one or more processors, the weight to be a second value, in response to determining that the gaze direction of the user is away from the virtual keyboard, the second value higher than the first value.

6. The method of claim 1 , wherein determining the weight of the language model includes:
determining, by the one or more processors, the weight to be a first value, in response to determining that the speed of the hand motion is less than a predetermined threshold, and
determining, by the one or more processors, the weight to be a second value, in response to determining that the speed of the hand motion is higher than the predetermined threshold, the second value higher than the first value.

7. The method of claim 1 , further comprising:
determining, by the one or more processors, a type of content corresponding to the input,
wherein the weight of the language model is determined according to the determined type of content.

8. The method of claim 1 , wherein modifying, by the one or more processors, the detected input according to the determined weight of the language model includes:
determining, by the one or more processors, a distribution of first characters in the detected input during a time period,
predicting, by the one or more processors via the language model according to the determined weight and the distribution of the first characters, semantically meaningful characters, one or more characters in the semantically meaningful characters different from one or more corresponding characters in the first characters, and
replacing the one or more corresponding characters with the one or more characters.

9. A device comprising:
at least one processor configured to:
determine a level of focus of a user when providing a hand motion with respect to a virtual keyboard, using data from at least one sensor including an image sensor communicably coupled to the at least one processor, the data indicative of a gaze or face direction of the user relative to a position of the virtual keyboard, wherein the at least processor is configured to determine the level of focus of the user when providing the hand motion by determining a speed of the hand motion using the data acquired by the image sensor,
detect, via the image sensor, an input to the virtual keyboard based on the hand motion,
determine weight of a language model according to the data indicative of the gaze or face direction of the user relative to the position of the virtual keyboard, and
modify the detected input according to the determined weight of the language model.

10. The device of claim 9 , wherein the at least one processor is configured to determine the level of focus of the user when providing the hand motion by determining an orientation of a head of the user.

11. The device of claim 9 , wherein the at least one processor is configured to determine the level of focus of the user when providing the hand motion by determining the gaze direction of the user.

12. The device of claim 9 , wherein the at least one processor is configured to determine a type of content corresponding to the input, wherein the at least one processor is configured to determine the weight of the language model according to the determined type of content.

13. The device of claim 9 , wherein the at least one processor is configured to modify the detected input according to the determined weight of the language model by:
determining a distribution of first characters in the detected input during a time period,
predicting, via the language model according to the determined weight and the distribution of the first characters, semantically meaningful characters, one or more characters in the semantically meaningful characters different from one or more corresponding characters in the first characters, and
replacing the one or more corresponding characters with the one or more characters.

14. A non-transitory computer readable medium storing instructions when executed by at least one processor cause the at least one processor to:
determine a level of focus of a user when providing a hand motion with respect to a virtual keyboard, using data from at least one sensor including an image sensor communicably coupled to the at least one processor, the data indicative of a gaze or face direction of the user relative to a position of the virtual keyboard, wherein the at least one processor is configured to determine the level of focus of the user when providing the hand motion by determining a speed of the hand motion using the data acquired by the image sensor;
detect, via the image sensor, an input to the virtual keyboard based on the hand motion;
determine weight of a language model according to the data indicative of the gaze or face direction of the user relative to the position of the virtual keyboard; and
modify the detected input according to the determined weight of the language model.

15. The non-transitory computer readable medium of claim 14 , wherein the instructions when executed by the at least one processor cause the at least one processor to determine the level of focus of the user when providing the hand motion by determining an orientation of a head of the user.

16. The non-transitory computer readable medium of claim 14 , wherein the instructions when executed by the at least one processor cause the at least one processor to determine the level of focus of

Assignees

• Facebook Tech Llc

Inventors

• Richardson Mark A
• Wang Robert Y

Classifications

• G06F3/04886 Primary

  by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus · CPC title

• G06F3/04842

  Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence) · CPC title

• G06F40/20

  Natural language analysis (semantic analysis of natural language G06F40/30) · CPC title

• G06F3/011

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• G06F3/0237

  using prediction or retrieval techniques · CPC title