Extended reality based digital assistant interactions

Patent metadata

Field	Value
Publication number	US-12423917-B2
Application number	US-202318202849-A
Country	US
Kind code	B2
Filing date	May 26, 2023
Priority date	Jun 10, 2022
Publication date	Sep 23, 2025
Grant date	Sep 23, 2025

Abstract

An example process includes: while displaying a portion of an extended reality (XR) environment representing a current field of view of a user: detecting a user gaze at a first object displayed in the XR environment, where the first object is persistent in the current field of view of the XR environment; in response to detecting the user gaze at the first object, expanding the first object into a list of objects including a second object representing a digital assistant; detecting a user gaze at the second object; in accordance with detecting the user gaze at the second object, displaying a first animation of the second object indicating that a digital assistant session is initiated; receiving a first audio input from the user; and displaying a second animation of the second object indicating that the digital assistant is actively listening to the user.

First claim

What is claimed is:

1. A non-transitory computer-readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by one or more processors of an electronic device with one or more sensors, cause the electronic device to:
while displaying a portion of an extended reality (XR) environment representing a current field of view of a user of the electronic device:
detect, with the one or more sensors, a user gaze at a first object displayed in the XR environment, wherein the first object is persistent in the current field of view of the XR environment;
in response to detecting the user gaze at the first object, expand the first object into a list of objects, wherein the list of objects includes a second object representing a digital assistant;
detect, with the one or more sensors, a user gaze at the second object;
in accordance with detecting the user gaze at the second object, display a first animation of the second object indicating that a first digital assistant session is initiated;

receive a first audio input from the user of the electronic device; and
display a second animation of the second object indicating that the digital assistant is actively listening to the user in response to receiving the first audio input, wherein the first animation is different from the second animation of the second object.

2. The non-transitory computer-readable storage medium of claim 1 , wherein displaying the first animation of the second object includes displaying a change in a shape, a size, or a color of the second object.

3. The non-transitory computer-readable storage medium of claim 1 , wherein displaying the first animation of the second object includes moving the second object away from the list of objects.

4. The non-transitory computer-readable storage medium of claim 1 , wherein displaying the first animation of the second object includes:
ceasing to display of the list of objects.

5. The non-transitory computer-readable storage medium of claim 1 , wherein displaying the first animation of the second object includes determining, based on the user gaze at the second object, that the first audio input is intended for the digital assistant.

6. The non-transitory computer-readable storage medium of claim 1 , wherein the one or more programs further comprise instructions, which when executed by the one or more processors, cause the electronic device to:
while displaying the list of objects, receive a hand gesture from the user, the hand gesture corresponding to a selection of the second object; and
in response to receiving the hand gesture, display an animation of the second object, wherein the animation of the second object indicates that a second digital assistant session is initiated.

7. The non-transitory computer-readable storage medium of claim 1 , wherein the one or more programs further comprise instructions, which when executed by the one or more processors, cause the electronic device to:
receive a second audio input including a spoken trigger for initiating a second digital assistant session; and
in response to receiving the second audio input, initiate the second digital assistant session, including displaying an animation of the second object.

8. The non-transitory computer-readable storage medium of claim 1 , wherein displaying the second animation of the second object includes expanding and shrinking a size of the second object responsive to the first audio input.

9. The non-transitory computer-readable storage medium of claim 1 , wherein the one or more programs further comprise instructions, which when executed by the one or more processors, cause the electronic device to:
display a virtual object in response to receiving the first audio input, the virtual object corresponding to a response, by the digital assistant, to the first audio input.

10. The non-transitory computer-readable storage medium of claim 9 , wherein the virtual object and the second object are persistent in the current field of view of the XR environment.

11. The non-transitory computer-readable storage medium of claim 9 , wherein the second object is persistent in the current field of view of the XR environment while the virtual object is positioned at a fixed location in the XR environment.

12. The non-transitory computer-readable storage medium of claim 9 , wherein the virtual object and the second object are positioned at respective fixed locations in the XR environment.

13. The non-transitory computer-readable storage medium of claim 9 , wherein the virtual object is displayed below the second object.

14. The non-transitory computer-readable storage medium of claim 9 , wherein the virtual object is positioned within a predetermined distance from the second object in the XR environment.

15. The non-transitory computer-readable storage medium of claim 9 , wherein the one or more programs further comprise instructions, which when executed by the one or more processors, cause the electronic device to:
receive, from the user, a request to interact with the virtual object; and
in response to receiving the request to interact with the virtual object, expand the virtual object into a user interface of an application corresponding to the virtual object.

16. The non-transitory computer-readable storage medium of claim 15 , wherein:
the request to interact with the virtual object corresponds to moving the virtual object from an initial location to a destination location; and
expanding the virtual object into the user interface includes displaying the user interface at the destination location.

17. The non-transitory computer-readable storage medium of claim 15 , wherein:
the second object is displayed while the user interface is displayed; and
the user can control the application using the second object by gazing at the second object or by speaking a trigger phrase.

18. The non-transitory computer-readable storage medium of claim 9 , wherein the one or more programs further comprise instructions, which when executed by the one or more processors, cause the electronic device to:
receive a request to integrate the virtual object into a second application; and
in response to receiving the request to integrate the virtual object into the second application, integrate the virtual object into the second application, including:
displaying a content of the virtual object within a user interface of the second application.

19. The non-transitory computer-readable storage medium of claim 18 , wherein:
while the content of the virtual object is displayed within the user interface of the second application:
the second object is displayed outside of the user interface of the second application.

20. The non-transitory computer-readable storage medium of claim 18 , wherein:
while the content of the virtual object is displayed within the user interface of the second application:
the second object is displayed inside of the user interface of the second application.

21. The non-transitory computer-readable storage medium of claim 9 , wherein the one or more programs further comprise instructions, which when executed by the one or more processors, cause the electronic device to:
in accordance with displaying the second object:
cease to display the second object a predetermined duration after displaying the virtual object; and
display the first object.

22. The non-transitory computer-readable storage medium of claim 9 ,

Assignees

• Apple Inc

Inventors

• Streja Lynn I
• Avery Keith P
• Daryanani Karan M
• Lemay Stephen O
• Lukens Myra C
• Maddika Sreeneel K
• Mannemala Chaitanya
• Manoharan Aswath
• Mari Pedro
• Moon Jay
• Rawat Abhishek

Classifications

• G06F3/017

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title

• G06T2219/2021

  Shape modification · CPC title

• G06T2219/2016

  Rotation, translation, scaling · CPC title

• G06T2219/2012

  Colour editing, changing, or manipulating; Use of colour codes · CPC title

• G10L2015/223

  Execution procedure of a spoken command · CPC title