Virtual keyboard engagement

Patent metadata

Field	Value
Publication number	US-11656762-B2
Application number	US-202117308476-A
Country	US
Kind code	B2
Filing date	May 5, 2021
Priority date	Jul 11, 2019
Publication date	May 23, 2023
Grant date	May 23, 2023

Abstract

The discussion relates to virtual keyboard engagement. One example can define key volumes relating to keys of a virtual keyboard and detect finger movement of a user through individual key volumes. The example can detect parameter changes associated with detected finger movement through individual key volumes and build potential key sequences from detected parameter changes.

First claim

The invention claimed is:

1. A device, comprising:
a display configured to be positioned on a head of a user in front of the user's eyes and configured to generate virtual content that appears to the user in a scene and that includes virtual keys;
sensors configured to monitor volumes defined at least in part by individual virtual keys and that extend both toward and away from the user relative to the virtual keys; and,
a processor configured to assign first probabilities to movement of a user extremity through individual volumes that is parallel to the virtual keys and dwell time of the user extremity is below a threshold and to assign second higher probabilities to user movement through individual volumes that is toward the user in the volume and where the user movement is away from the user in the volume and the dwell time of the user extremity is above the threshold and to predict an intended user action relative to the virtual content utilizing the first probabilities and the second probabilities.

2. The device of claim 1 , wherein the display is configured to generate the virtual content so that the virtual keys appear to be projected into 3D space from a perspective of the user.

3. The device of claim 1 , wherein the sensors are configured to detect parameters associated with the movement of the user extremity through the volumes and wherein the parameters are utilized by the processor to assign the first probabilities and the second higher probabilities.

4. The device of claim 3 , wherein the parameters further include change in direction and change in velocity relative to individual volumes.

5. The device of claim 4 , wherein the virtual content is associated with individual virtual keys associated with characters, and wherein the processor is configured to compare a sequence of characters associated with the movement to a dictionary of known words to predict the intended user action as spelling an individual word from the dictionary.

6. The device of claim 5 , wherein the processor is configured to perform the intended user action.

7. The device of claim 6 , wherein the processor is configured to perform the intended user action by auto-populating the individual word into a dialog box or providing the individual word as an auto-suggestion for the dialog box.

8. The device of claim 7 , wherein the processor is configured to perform the auto-populating when a confidence associated with the individual word exceeds a threshold.

9. A device-implemented method, comprising:
generating a virtual keyboard with head mounted optics positioned in front of eyes of a user;
detecting a path of a user extremity in 3D space proximate to a perceived position of the virtual keyboard;
assigning probabilities to individual keys of the virtual keyboard when the path passes through a volume defined at least in part by the individual keys and extending toward and away from the user relative to the individual keys;
updating the probabilities relative to an individual key to a higher probability when the path relative to an individual key entails a dwell time of the user extremity above a threshold and includes directional changes including movement toward the user in the volume relative to the individual key and away from the user in the volume relative to the individual key;
predicting an intended user action based at least in part on the probabilities and the updated probabilities; and,
automatically taking an action on behalf of the user based at least in part upon the predicted intended user action.

10. The device-implemented method of claim 9 , wherein detecting a path of a user extremity comprises detecting the path of a finger, multiple fingers, a hand, fingers of both a left and right hand, and/or an object held in the hand.

11. The device-implemented method of claim 9 , wherein the updating comprises assigning highest probabilities to an individual key where the path through the individual key includes movement parallel to the individual key and either toward or away from the user in the volume relative to the individual key and the dwell time relative to the individual key compared to other individual keys.

12. The device-implemented method of claim 9 , wherein assigning probabilities to individual keys of the virtual keyboard comprises assigning a first probability to an individual key of the keyboard when the path passes relative to the individual key under first parameters and assigning a second higher probability to another individual key when the path passes relative to the another individual key under second parameters.

13. The device-implemented method of claim 12 , wherein taking an action comprises populating a dialog box with a word selected based at least in part upon the at least one of the first probability, the second higher probability, and/or the updated probabilities.

14. A device-implemented method, comprising:
generating virtual content with a device positioned on a head of a user, the virtual content including virtual keys and appearing to the user in a volume beyond the device;
defining virtual content volumes that extend both in front of and behind individual virtual keys relative to the user,
detecting finger movement of the user through individual virtual content volumes;
detecting parameter changes associated with detected finger movement through individual virtual content volumes, the parameter changes including changes in direction relative to a surface of an individual key and changes in velocity relative to the individual key; and,
building potential user selections of individual virtual keys from detected parameter changes at least in part by assigning probabilities based on the detected parameter changes.

15. The device-implemented method of claim 14 , wherein generating virtual content comprises generating a virtual keyboard and the defining virtual content volumes are defined at least in part by eyes of the user and a periphery of individual virtual keys or wherein the virtual content volumes are defined as cuboids that extend both in front of and behind individual keys relative to the user.

16. The device-implemented method of claim 15 , wherein detecting parameter changes further comprises detecting changes in direction of the finger movement both towards and away from an individual virtual key.

17. The device-implemented method of claim 16 , wherein detecting parameter changes in direction comprises at least one of: detecting changes in direction parallel to a key surface and detecting changes in a direction perpendicular to the key surface.

18. The device-implemented method of claim 17 , wherein the detecting parameter changes further comprises assigning higher probabilities to individual virtual keys in potential key sequences associated with dwell times of finger movement for individual virtual content volumes above a threshold.

19. The device-implemented method of claim 18 , further comprising comparing the potential key sequences to known words.

20. The device-implemented method of claim 18 , further comprising comparing the potential key sequences to known user commands.

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Becker Christopher M
• Eldirghami Nazeeh A
• Barnes Kevin W
• Schwarz Julia
• Carter Eric

Classifications

• G06F3/04886 Primary

  by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus · CPC title

• G06F3/017

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title

• G06F3/011 Primary

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title