Virtual keyboard

Patent metadata

Field	Value
Publication number	US-11270515-B2
Application number	US-202017010237-A
Country	US
Kind code	B2
Filing date	Sep 2, 2020
Priority date	Sep 4, 2019
Publication date	Mar 8, 2022
Grant date	Mar 8, 2022

Abstract

Systems, apparatuses (or devices), methods, and computer-readable media are provided for generating virtual content. For example, a device (e.g., an extended reality device) can obtain an image of a scene of a real-world environment, wherein the real-world environment is viewable through a display of the extended reality device as virtual content is displayed by the display. The device can detect at least a part of a physical hand of a user in the image. The device can generate a virtual keyboard based on detecting at least the part of the physical hand. The device can determine a position for the virtual keyboard on the display of the extended reality device relative to at least the part of the physical hand. The device can display the virtual keyboard at the position on the display.

First claim

What is claimed is:

1. A method of generating virtual content, the method comprising:
obtaining, by an extended reality device, an image of a scene of a real-world environment, wherein the real-world environment is viewable through a display of the extended reality device as virtual content is displayed by the display;
detecting, by the extended reality device, at least a part of a physical hand of a user in the image;
generating, in response to detecting at least the part of the physical hand, a virtual keyboard;
determining a position for the virtual keyboard on the display of the extended reality device, the position being determined relative to at least the part of the physical hand; and
displaying, by the extended reality device, the virtual keyboard at the position on the display.

2. The method of claim 1 , further comprising:
detecting one or more landmark points on the physical hand;
determining one or more locations of the one or more landmark points with respect to a camera used to capture the image; and
determining, based on the one or more locations of the one or more landmark points with respect to the camera, the position for the virtual keyboard on the display relative to the physical hand.

3. The method of claim 1 , further comprising:
determining a pose of a head of the user; and
determining, based on the pose of the head, the position for the virtual keyboard on the display relative to the head.

4. The method of claim 1 , wherein the virtual keyboard is fixed at the position on the display as the physical hand moves positions.

5. The method of claim 1 , further comprising:
receiving input associated with operation of the virtual keyboard; and
maintaining the virtual keyboard at the position as the virtual keyboard is operated based on the received input.

6. The method of claim 1 , further comprising:
determining at least the part of the physical hand is in a different location in an additional image of the scene as compared to a location of at least the part of the physical hand in the image; and
displaying, based on determining at least the part of the physical hand is in the different location in the additional image of the scene, the virtual keyboard at an additional position on the display, the additional position being different than the position.

7. The method of claim 1 , further comprising:
detecting expiration of a predetermined amount of time after determining the position for the virtual keyboard on the display; and
displaying, based on detecting expiration of the predetermined amount of time, the virtual keyboard at an additional position on the display, the additional position being different than the position.

8. The method of claim 1 , wherein at least the part of the physical hand includes at least one point on the physical hand and at least one point on a finger of the physical hand.

9. The method of claim 1 , wherein at least the part of the physical hand includes at least one point on the physical hand and at least one point on each finger of the physical hand.

10. The method of claim 1 , wherein the virtual keyboard includes a first part and a second part, the first part being displayed at the position on the display relative to the physical hand, and the second part being displayed at an additional position on the display relative to an additional physical hand of the user.

11. The method of claim 10 , wherein the first part of the virtual keyboard moves on the display relative to the physical hand, and wherein the second part of the virtual keyboard moves on the display relative to the additional physical hand.

12. The method of claim 1 , further comprising:
determining the physical hand is not present in an additional image of the scene; and
removing, based on determining the physical hand is not present in the additional image of the scene, the virtual keyboard from the display.

13. The method of claim 12 , wherein the virtual keyboard is active for providing input when the virtual keyboard is removed from the display.

14. The method of claim 13 , further comprising deactivating the virtual keyboard from being used for providing input.

15. The method of claim 1 , wherein the virtual keyboard is displayed when the physical hand is not present in one or more images of the scene.

16. An apparatus for generating virtual content, comprising:
a display;
a memory configured to store one or more images; and
a processor implemented in circuitry and coupled to the memory, the processor being configured to:
obtain an image of a scene of a real-world environment, wherein the real-world environment is viewable through a display of the apparatus as virtual content is displayed by the display;
detect at least a part of a physical hand of a user in the image;
generate, in response to detecting at least the part of the physical hand, a virtual keyboard;
determine a position for the virtual keyboard on the display, the position being determined relative to at least the part of the physical hand; and
display the virtual keyboard at the position on the display.

17. The apparatus of claim 16 , wherein the processor is further configured to:
detect one or more landmark points on the physical hand;
determine one or more locations of the one or more landmark points with respect to a camera used to capture the image; and
determine, based on the one or more locations of the one or more landmark points with respect to the camera, the position for the virtual keyboard on the display relative to the physical hand.

18. The apparatus of claim 16 , wherein the processor is further configured to:
determine a pose of a head of the user; and
determine, based on the pose of the head, the position for the virtual keyboard on the display relative to the head.

19. The apparatus of claim 16 , wherein the virtual keyboard is fixed at the position on the display as the physical hand moves positions.

20. The apparatus of claim 16 , wherein the processor is further configured to:
receive input associated with operation of the virtual keyboard; and
maintain the virtual keyboard at the position as the virtual keyboard is operated based on the received input.

21. The apparatus of claim 16 , wherein the processor is further configured to:
determine at least the part of the physical hand is in a different location in an additional image of the scene as compared to a location of at least the part of the physical hand in the image; and
display, based on determining at least the part of the physical hand is in the different location in the additional image of the scene, the virtual keyboard at an additional position on the display, the additional position being different than the position.

22. The apparatus of claim 16 , wherein the processor is further configured to:
detect expiration of a predetermined amount of time after determining the position for the virtual keyboard on the display; and
display, based on detecting expiration of the predetermined amount of time, the virtual keyboard at an additional position on the display, the additional position being different than the position.

23. The apparatus of claim 16 , wherein at least the part of the physical hand includes at least one point on the physical hand and at least one point on a finger of the physical hand.

24. The apparatus of claim 16 , wherein the virtual keyboard includes a first part and a second part, the first part b

Assignees

• Qualcomm Inc

Inventors

• Beith Scott
• Kies Jonathan
• Tartz Robert
• Kandhadai Ananthapadmanabhan Arasanipalai

Classifications

• G06F3/017

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title

• G06F1/163

  Wearable computers, e.g. on a belt · CPC title

• G06F3/0426

  tracking fingers with respect to a virtual keyboard projected or printed on the surface (virtual keyboards on touch screens G06F3/04886) · CPC title

• G06F3/012

  Head tracking input arrangements · CPC title

• G06F3/0304

  Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042) · CPC title