Dynamic configuration of user interface layouts and inputs for extended reality systems

Patent metadata

Field	Value
Publication number	US-12399568-B2
Application number	US-202117481029-A
Country	US
Kind code	B2
Filing date	Sep 21, 2021
Priority date	Sep 30, 2020
Publication date	Aug 26, 2025
Grant date	Aug 26, 2025

Abstract

Systems and techniques are provided for generating virtual content. For example, a process can include obtaining an image of a physical object in a real-world environment and determining at least one attribute of the physical object in the image. The process can include determining an interface layout for virtual content based on the at least one attribute of the physical object. The process can include displaying the virtual content based on the determined interface layout. In some cases, the process can including determining an input type based on the at least one attribute of the physical object, receiving input corresponding to the input type, and controlling displayed virtual content based on the determined input type.

First claim

What is claimed is:

1. An apparatus for generating virtual content, comprising:
a memory; and
one or more processors coupled to the memory and configured to:
obtain, from an image sensor, an image of a physical object in a real-world environment;
determine, from the image, at least one attribute of the physical object in the image obtained from the image sensor;
determine, from the image, a position of a first hand of a user of the apparatus holding the physical object;
determine an interface layout for virtual content based on the at least one attribute of the physical object determined in the image and the position of the first hand holding the physical object;
determine an input type for the virtual content and an input function for the input type based on the at least one attribute of the physical object determined in the image and the position of the first hand holding the physical object, the input type defining a scroll input and the input function defining a vertical direction for the scroll input;
receive a user input, the user input comprising a scroll input in the vertical direction;
cause the virtual content to be displayed relative to the physical object based on the determined interface layout and manipulated in the vertical direction based on the received user input;
determine, from an additional image, the first hand of the user holding the physical object and a second hand of the user within a threshold distance of the virtual content displayed relative to the physical object; and
determine at least one of an updated input type for controlling the virtual content or an updated input function for the input type based on determining the first hand of the user holding the physical object and the second hand of the user within the threshold distance of the virtual content, the updated input type defining a tap-based input and the updated input function defining a horizontal direction for the scroll input.

2. The apparatus of claim 1 , wherein the real-world environment is viewable through a display of the apparatus as the virtual content is displayed by the display.

3. The apparatus of claim 1 , wherein the at least one attribute of the physical object includes at least one of a size of the physical object, a shape of the physical object, an orientation of the physical object with respect to the apparatus, a position of the physical object with respect to the apparatus, or an ability of the physical object to be held by a hand of a user.

4. The apparatus of claim 1 , wherein the one or more processors are configured to:
determine a body part of a user in one or more images;
determine at least one attribute of the body part of the user in the one or more images; and
determine the interface layout for the virtual content based on the at least one attribute of the physical object and the at least one attribute of the body part of the user.

5. The apparatus of claim 4 , wherein the at least one attribute of the body part includes at least one of a size of the body part, an orientation of the body part relative to the physical object, a position of the body part relative to the physical object, an orientation of the body part relative to the apparatus, or a position of the body part relative to the apparatus.

6. The apparatus of claim 4 , wherein the body part of the user is the first hand of the user, and wherein the at least one attribute of the body part includes at least one of a size of the first hand, an orientation of the first hand relative to the physical object, an orientation of the first hand relative to the apparatus, a position of the first hand relative to the apparatus, or whether the first hand is a left hand or a right hand of the user.

7. The apparatus of claim 1 , wherein the one or more processors are configured to:
determine whether the first hand is a left hand or a right hand of the user; and
determine a position for the virtual content in the interface layout based on whether the first hand is the left hand or the right hand of the user.

8. The apparatus of claim 7 , wherein the one or more processors are configured to:
determine the first hand is the left hand of the user;
determine, based on the first hand being the left hand of the user, the position for the virtual content relative to a right side of the physical object; and
display the virtual content in the interface layout relative to the right side of the physical object.

9. The apparatus of claim 7 , wherein the one or more processors are configured to:
determine the first hand is the right hand of the user;
determine, based on the first hand being the right hand of the user, the position for the virtual content relative to a left side of the physical object; and
display the virtual content in the interface layout relative to the left side of the physical object.

10. The apparatus of claim 1 , wherein the one or more processors are configured to:
determine the at least one attribute of the physical object in a plurality of images.

11. The apparatus of claim 1 , wherein the apparatus comprises an extended reality device.

12. The apparatus of claim 1 , further comprising a display configured to display the virtual content.

13. The apparatus of claim 1 , further comprising a camera configured to capture one or more images.

14. A method of generating virtual content, the method comprising:
obtaining, by a device from an image sensor, an image of a physical object in a real-world environment;
determining, by the device from the image, at least one attribute of the physical object in the image obtained from the image sensor;
determining, from the image, a position of a first hand of a user of the device holding the physical object;
determining an interface layout for virtual content based on the at least one attribute of the physical object determined in the image and the position of the first hand holding the physical object;
determining an input type for the virtual content and an input function for the input type based on the at least one attribute of the physical object determined in the image and the position of the first hand holding the physical object, the input type defining a scroll input and the input function defining a vertical direction for the scroll input;
receiving a user input, the user input comprising a scroll input in the vertical direction;
displaying the virtual content relative to the physical object based on the determined interface layout;
manipulating the virtual content in the vertical direction based on the received user input;
determining, from an additional image, the first hand of the user holding the physical object and a second hand of the user within a threshold distance of the virtual content displayed relative to the physical object; and
determining at least one of an updated input type for controlling the virtual content or an updated input function for the input type based on determining the first hand of the user holding the physical object and the second hand of the user within the threshold distance of the virtual content, the updated input type defining a tap-based input and the updated input function defining a horizontal direction for the scroll input.

15. The method of claim 14 , wherein the real-world environment is viewable through a display of the device as the virtual content is displayed by the display.

16. The method of claim 14 , wherein the at least one attribute of the physical object includes at least one of a size of the physical object, a shape of the physical object, an orientation of the physical object with r

Assignees

• Qualcomm Inc

Inventors

• Kies Jonathan
• Willkie Chad

Classifications

• G06V20/20

  in augmented reality scenes · CPC title

• G06V40/107

  Static hand or arm · CPC title

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06F3/0485

  Scrolling or panning · CPC title

• G06F3/04815

  Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object · CPC title