Virtual content generation

Patent metadata

Field	Value
Publication number	US-12347052-B2
Application number	US-202418666374-A
Country	US
Kind code	B2
Filing date	May 16, 2024
Priority date	Sep 4, 2019
Publication date	Jul 1, 2025
Grant date	Jul 1, 2025

Abstract

Systems, apparatuses (or devices), methods, and computer-readable media are provided for generating virtual content. For example, a device (e.g., an extended reality device) can obtain an image of a scene of a real-world environment, wherein the real-world environment is viewable through a display of the extended reality device as virtual content is displayed by the display. The device can detect at least a part of a physical hand of a user in the image. The device can generate a virtual keyboard based on detecting at least the part of the physical hand. The device can determine a position for the virtual keyboard on the display of the extended reality device relative to at least the part of the physical hand. The device can display the virtual keyboard at the position on the display.

First claim

What is claimed is:

1. An apparatus for generating virtual content, the apparatus comprising:
at least one memory; and
at least one processor configured in circuitry and coupled to the at least one memory, the at least one processor being configured to:
obtain an image of a scene of a real-world environment, wherein the real-world environment viewable through a display of the apparatus as virtual content is displayed by the display;
detect at least a pose of a hand of a user in the image;
generate, in response to detecting at least the pose of the hand, a graphical user interface (GUI) element;
determine a position for the GUI element on the display of the apparatus, the position being determined relative to at least the pose of the hand; and
display the GUI element at the position on the display.

2. The apparatus of claim 1 , wherein the position of the GUI element is adjustable by the user.

3. The apparatus of claim 1 , wherein the at least one processor is configured to:
detect one or more landmark points on the hand;
determine one or more locations of the one or more landmark points with respect to a camera used to capture the image; and
determine, based on the one or more locations of the one or more landmark points with respect to the camera, the position for the GUI element on the display relative to the hand.

4. The apparatus of claim 1 , wherein the at least one processor is configured to:
determine a pose of a head of the user; and
determine, based on the pose of the head, the position for the GUI element on the display relative to the head.

5. The apparatus of claim 1 , wherein the GUI element is fixed at the position on the display as the hand moves positions.

6. The apparatus of claim 1 , wherein the at least one processor is configured to:
receive input associated with operation of the GUI element; and
maintain the GUI element at the position as the GUI element is operated based on the received input.

7. The apparatus of claim 1 , wherein the at least one processor is configured to:
determine at least the pose of the hand is in a different location in an additional image of the scene as compared to a location of at least the pose of the hand in the image; and
display, based on determining at least the pose of the hand is in the different location in the additional image of the scene, the GUI element at an additional position on the display, the additional position being different than the position.

8. The apparatus of claim 1 , wherein the at least one processor is configured to:
detect expiration of a predetermined amount of time after determining the position for the GUI element on the display; and
display, based on detecting expiration of the predetermined amount of time, the GUI element at an additional position on the display, the additional position being different than the position.

9. The apparatus of claim 1 , wherein at least the pose of the hand includes at least one point on the hand and at least one point on a finger of the hand.

10. The apparatus of claim 1 , wherein at least the pose of the hand includes at least one point on the hand and at least one point on each finger of the hand.

11. The apparatus of claim 1 , wherein the GUI element includes a first part and a second part, the first part being displayed at the position on the display relative to the hand, and the second part being displayed at an additional position on the display relative to an additional hand of the user.

12. The apparatus of claim 11 , wherein the first part of the GUI element moves on the display relative to the hand, and wherein the second part of the GUI element moves on the display relative to the additional hand.

13. The apparatus of claim 1 , wherein the at least one processor is configured to:
determine the hand is not present in an additional image of the scene; and
remove, based on determining the hand is not present in the additional image of the scene, the GUI element from the display.

14. The apparatus of claim 13 , wherein the GUI element is active for providing input when the GUI element is removed from the display.

15. The apparatus of claim 14 , wherein the at least one processor is configured to deactivate the GUI element from being used for providing input.

16. The apparatus of claim 1 , wherein the at least one processor is configured to display the GUI element when the hand is not present in one or more images of the scene.

17. The apparatus of claim 1 , wherein the apparatus comprises an extended reality device including a display and at least one camera configured to capture one or more images of the scene.

18. A method of generating virtual content, the method comprising:
obtaining, by an extended reality device, an image of a scene of a real-world environment, wherein the real-world environment viewable through a display of the extended reality device as virtual content is displayed by the display;
detecting, by the extended reality device, at least a pose of a hand of a user in the image;
generating, in response to detecting at least the pose of the hand, a graphical user interface (GUI) element;
determining a position for the GUI element on the display of the extended reality device, the position being determined relative to at least the pose of the hand; and
displaying, by the extended reality device, the GUI element at the position on the display.

19. The method of claim 18 , wherein the position of the GUI element is adjustable by the user.

20. The method of claim 18 , further comprising:
detecting one or more landmark points on the hand;
determining one or more locations of the one or more landmark points with respect to a camera used to capture the image; and
determining, based on the one or more locations of the one or more landmark points with respect to the camera, the position for the GUI element on the display relative to the hand.

Assignees

• Qualcomm Inc

Inventors

• Beith Scott
• Kies Jonathan
• Tartz Robert
• Kandhadai Ananthapadmanabhan Arasanipalai

Classifications

• G06F3/017

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title

• G06F2203/04803

  Split screen, i.e. subdividing the display area or the window area into separate subareas · CPC title

• G02B27/0172

  characterised by optical features · CPC title

• G02B2027/0138

  comprising image capture systems, e.g. camera · CPC title

• G06F3/011 Primary

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title