Devices, methods, and graphical user interfaces for interacting with three-dimensional environments

Patent metadata

Field	Value
Publication number	US-12314529-B2
Application number	US-202318244864-A
Country	US
Kind code	B2
Filing date	Sep 11, 2023
Priority date	Sep 27, 2019
Publication date	May 27, 2025
Grant date	May 27, 2025

Abstract

An electronic device, while displaying a three-dimensional environment, including one or more virtual objects, detects gaze of a user directed toward a first virtual object in the three-dimensional environment. The gaze meets first criteria and the first virtual object is responsive to at least one gesture input. In response to detecting the gaze that meets the first criteria, in accordance with a determination that a hand is in a predefined ready state for providing gesture inputs, the electronic device displays an indication of one or more interaction options available for the first virtual object in the three-dimensional environment; and in accordance with a determination the hand is not in the predefined ready state for providing gesture inputs, the electronic device forgoes displaying the indication of one or more interaction options available for the first virtual object.

First claim

What is claimed is:

1. A method, comprising:
at an electronic device including a display generation component and one or more input devices:
displaying, via the display generation component, a three-dimensional environment, including one or more virtual objects;
detecting, via the one or more input devices, gaze, of a user of the electronic device, directed toward a first virtual object in the three-dimensional environment, wherein the gaze meets first criteria and the first virtual object is responsive to at least one gesture input; and
in response to detecting the gaze that meets the first criteria and that is directed toward the first virtual object responsive to at least one gesture input:
in accordance with a determination that a hand is in a predefined ready state for providing gesture inputs while detecting that the gaze of the user is directed toward the first virtual object, displaying, via the display generation component, an indication of one or more interaction options available for the first virtual object in the three-dimensional environment; and
in accordance with a determination the hand is not in the predefined ready state for providing gesture inputs while detecting that the gaze of the user is directed toward the first virtual object, forgoing displaying the indication of one or more interaction options available for the first virtual object.

2. The method of claim 1 , wherein determining that the hand is in the predefined ready state for providing gesture inputs includes determining that a predefined portion of the hand is in contact with a physical control element.

3. The method of claim 1 , wherein determining that the hand is in the predefined ready state for providing gesture inputs includes determining that the hand is raised above a predefined level relative to the user.

4. The method of claim 1 , wherein determining that the hand is in the predefined ready state for providing gesture inputs includes determining that the hand is in a predefined configuration.

5. The method of claim 1 , wherein displaying the indication of one or more interaction options available for the first virtual object includes displaying information about the first virtual object that is adjustable in response to a subsequent input.

6. The method of claim 1 , wherein displaying the indication of one or more interaction options available for the first virtual object includes displaying an animation of the first virtual object.

7. The method of claim 1 , wherein displaying the indication of one or more interaction options available for the first virtual object includes displaying a selection indicator over at least a portion of the first virtual object.

8. The method of claim 1 , including:
detecting the gaze directed toward a second virtual object in the three-dimensional environment, wherein the gaze meets the first criteria and the second virtual object is responsive to at least one gesture input; and
in response to detecting the gaze that meets the first criteria and that is toward the second virtual object that is responsive to at least one gesture input:
in accordance with a determination that the hand is in the predefined ready state for providing gesture inputs, displaying an indication of one or more interaction options available for the second virtual object.

9. The method of claim 1 , including:
in response to detecting the gaze that meets the first criteria and that is directed toward the first virtual object that is responsive to at least one gesture input:
in accordance with the determination that the hand is in the predefined ready state for providing gesture inputs while detecting that the gaze of the user is directed toward the first virtual object:
detecting, via the one or more input devices, a first gesture input by the hand; and
in response to detecting the first gesture input by the hand, performing an interaction with the first virtual object.

10. The method of claim 9 , wherein the first virtual object includes a first image, and performing the interaction with the first virtual object includes replacing the first image with a second image that is different from the first image.

11. The method of claim 9 , wherein the first virtual object includes a first playable media content, and performing the interaction with the first virtual object includes toggling playback of the first playable media content.

12. The method of claim 9 , wherein the first virtual object is a virtual window displaying first virtual scenery, and performing the interaction with the first virtual object includes replacing display of the first virtual scenery with display of second virtual scenery that is different from the first virtual scenery.

13. The method of claim 9 , wherein:
the first gesture input is an upward flick gesture; and
performing the interaction with the first virtual object includes displaying a user interface with one or more interaction options for the first virtual object.

14. The method of claim 9 , wherein:
the first gesture input includes rotation of the hand; and
performing the interaction with the first virtual object includes changing an output volume of content associated with the first virtual object.

15. The method of claim 1 , wherein the first criteria include a requirement that the gaze remains directed toward the first virtual object for at least a threshold amount of time.

16. The method of claim 1 , wherein:
in accordance with a determination the hand is in the predefined ready state for providing gesture inputs while detecting that the gaze of the user is not directed toward the first virtual object, forgoing displaying the indication of one or more interaction options available for the first virtual object.

17. An electronic device, comprising:
one or more processors;
a display generation component;
one or more input devices; and
memory storing one or more programs that are configured for execution by the one or more processors, the one or more programs including instructions for:
displaying, via the display generation component, a three-dimensional environment, including one or more virtual objects;
detecting, via the one or more input devices, gaze, of a user of the electronic device, directed toward a first virtual object in the three-dimensional environment, wherein the gaze meets first criteria and the first virtual object is responsive to at least one gesture input; and
in response to detecting the gaze that meets the first criteria and that is toward the first virtual object responsive to at least one gesture input:
in accordance with a determination that a hand is in a predefined ready state for providing gesture inputs while detecting that the gaze of the user is directed toward the first virtual object, displaying, via the display generation component, an indication of one or more interaction options available for the first virtual object in the three-dimensional environment; and
in accordance with a determination the hand is not in the predefined ready state for providing gesture inputs while detecting that the gaze of the user is directed toward the first virtual object, forgoing displaying the indication of one or more interaction options available for the first virtual object.

18. The electronic device of claim 17 , wherein determining that the hand is in the predefined ready state for providing gesture inputs includes determining that a predefined portion of the hand is in contact with a physical control element.

19. The electronic device of claim 17 , w

Assignees

• Apple Inc

Inventors

• Lemay Stephen O
• Faulkner Jeffrey M
• Pastrana Vicente Israel
• Rockel Philipp
• Tilton Robert T
• Sorrentino Iii William A
• Bauerly Kristi E S
• Anton Peter D
• Dye Alan C

Classifications

• G06F3/017 Primary

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title

• G06T15/00

  Three-dimensional [3D] image rendering · CPC title

• G06T7/70

  Determining position or orientation of objects or cameras (camera calibration G06T7/80) · CPC title

• G06T2200/24

  involving graphical user interfaces [GUIs] · CPC title

• G06F3/04817

  using icons (graphical or visual programming using iconic symbols G06F8/34) · CPC title