Devices, methods, and graphical user interfaces for interacting with three-dimensional environments

Patent metadata

Field	Value
Publication number	US-11567625-B2
Application number	US-202117483717-A
Country	US
Kind code	B2
Filing date	Sep 23, 2021
Priority date	Sep 24, 2020
Publication date	Jan 31, 2023
Grant date	Jan 31, 2023

Abstract

A computing system, while displaying a first view of a first computer-generated three-dimensional environment including a representation of a respective portion of a physical environment, and a first representation of one or more projections of light in a first portion of the first computer-generated three-dimensional environment, detects, from a first user, a query directed to a virtual assistant. In response, the computer system displays animated changes of the first representation of the one or more projections of light in the first portion of the first computer-generated three-dimensional environment, including displaying a second representation of the one or more projections of light that is focused on a first sub-portion of the first portion, and then displays content responding to the query at a position corresponding to the first sub-portion of the first portion.

First claim

What is claimed is:

1. A method, comprising:
at a computing system including a first display generation component and one or more input devices:
displaying, via the first display generation component:
a first view of a first computer-generated three-dimensional environment, wherein the first view of the first computer-generated three-dimensional environment includes a representation of a respective portion of a physical environment, and
a first representation of one or more projections of light in a first portion of the first computer-generated three-dimensional environment, wherein the first representation of the one or more projections of light has an appearance that indicates a spatial relationship between a virtual assistant present in the first computer-generated three-dimensional environment and the representation of the respective portion of the physical environment in the first computer-generated three-dimensional environment;

while displaying the first view of the first computer-generated three-dimensional environment and the first representation of the one or more projections of light, detecting, from a first user, a query directed to the virtual assistant; and
in response to detecting the query directed to the virtual assistant:
displaying animated changes of the first representation of the one or more projections of light in the first portion of the first computer-generated three-dimensional environment, wherein displaying the animated changes include displaying a second representation of the one or more projections of light that are focused on a first sub-portion of the first portion of the first computer-generated three-dimensional environment; and
after displaying the animated changes of the first representation of the one or more projections of light, displaying content responding to the query at a position corresponding to the first sub-portion of the first portion of the first computer-generated three-dimensional environment.

2. The method of claim 1 , including:
while detecting the query directed to the virtual assistant, displaying continuous animation of the first representation of the one or more projections of light, wherein displaying the continuous animation includes displaying alterations of a respective representation of at least one projection of light in the one or more projections of light at a position corresponding to a second sub-portion of the first portion of the first computer-generated three-dimensional environment.

3. The method of claim 2 , wherein at least one of the first sub-portion and the second sub-portion of the first computer-generated three-dimensional environment includes a representation of a first surface in the respective portion of the physical environment, and displaying the continuous animation includes displaying alterations of the respective representation of at least one projection of light in the one or more projections of light at a position that corresponds to the representation of the first surface in the respective portion of the physical environment.

4. The method of claim 1 , wherein the first sub-portion of the first computer-generated three-dimensional environment includes a representation of a second surface in the respective portion of the physical environment, and displaying the content responding to the query at the position corresponding to the first sub-portion of the first portion of the first computer-generated three-dimensional environment includes displaying the content responding to the query at a position corresponding to the representation of the second surface in the respective portion of the physical environment.

5. The method of claim 1 , wherein displaying the animated changes include moving a source of the one or more projections of light in the first computer-generated three-dimensional environment toward the first sub-portion of the first portion of the first computer-generated three-dimensional environment.

6. The method of claim 5 , wherein moving the source of the one or more projections of light toward the first sub-portion of the first portion of the first computer-generated three-dimensional environment includes:
at a first time before detecting the query directed to the virtual assistant, displaying the first representation of the one or more projections of light without displaying a user interface object corresponding to the virtual assistant within the first view of the first computer-generated three-dimensional environment including the representation of the respective portion of the physical environment; and
at a second time after detecting the query directed to the virtual assistant, displaying the user interface object corresponding to the virtual assistant within the representation of the respective portion of the physical environment and displaying the first representation of the one or more projections of light extending outward from the user interface object corresponding to the virtual assistant.

7. The method of claim 1 , including:
detecting, by the one or more input devices, a first input that meets first criteria associated with identifying a location of attention of the first user providing the query, wherein the first sub-portion of the first portion of the first computer-generated three-dimensional environment onto which the one more projections of light are focused is selected by the computer system in accordance with the first input that meets the first criteria associated with identifying the location of attention of the first user providing the query.

8. The method of claim 7 , wherein the first input that meets the first criteria is a gaze input that is detected in conjunction with detection of the query, and wherein the gaze input meets preset stability criteria at a position in the first computer-generated three-dimensional environment that corresponds to a location of a physical object or surface in the respective portion of the physical environment.

9. The method of claim 7 , wherein the first input that meets the first criteria is a gesture input that is detected in conjunction with detection of the query, and wherein the gesture input is directed to a position in the first computer-generated three-dimensional environment that corresponds to a location of a physical object or surface in the respective portion of the physical environment.

10. The method of claim 1 , wherein the content responding to the query includes a representation of a second user that is different from the first user, and the method includes:
displaying, concurrently with the representation of the second user, a first visual indication that the second user is available to interact with the first user in the first computer-generated three-dimensional environment.

11. The method of claim 10 , including:
while displaying the first visual indication in the first view of the first computer-generated three-dimensional environment, detecting, by the one or more input devices, a second input that is directed to the representation of the second user; and
in response to detecting the second input that is directed to the representation of the second user and in accordance with a determination that the second input meets second criteria, initiating a shared computer-generated experience between the first user and the second user.

12. The method of claim 1 , including:
displaying a user interface object corresponding to the virtual assistant with a respective one of a plurality of appearances in the first view of the first computer-generated three-dimensional environment, wherein the plurality of appearances correspond to a plurality of different states of the virtual assistant in relation t

Assignees

• Apple Inc

Inventors

• Faulkner Jeffrey M
• Pastrana Vicente Israel
• Rockel Philipp
• Holder Wesley M
• Henderson Nicholas W
• Tilton Robert T
• Dellinger Richard R
• Lemay Stephen O
• Sorrentino Iii William A
• Yerkes Giancarlo
• Mari Pedro

Classifications

• G06F3/013

  Eye tracking input arrangements (G06F3/015 takes precedence) · CPC title

• G06F3/017

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title

• G06F3/04815 Primary

  Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object · CPC title

• G06F3/167

  Audio in a user interface, e.g. using voice commands for navigating, audio feedback · CPC title

• G06T15/506

  Illumination models · CPC title