Dynamically morphing virtual assistant avatars for assistant systems

Patent metadata

Field	Value
Publication number	US-12353897-B2
Application number	US-202217877568-A
Country	US
Kind code	B2
Filing date	Jul 29, 2022
Priority date	Jul 29, 2022
Publication date	Jul 8, 2025
Grant date	Jul 8, 2025

Abstract

In one embodiment, a method includes rendering for displays of an extended-reality (XR) display device a first output image of an XR assistant avatar within an environment in a field of view of a first user of the XR display device, wherein the XR assistant avatar has a first form and a first pose with respect to the first user and is interactable by the first user to access an assistant system, detecting a change in a context of the first user with respect to the environment, determining whether to morph the XR assistant avatar based on the detected change in the context of the first user, and rendering a second output image of the XR assistant avatar for the displays of the XR display device, wherein the XR assistant avatar is morphed to have a second form and a second pose with respect to the first user.

First claim

What is claimed is:

1. A method comprising, by a head-mounted extended-reality (XR) system:
rendering, for a display of the head-mounted XR system, a first output image of an XR assistant avatar within a virtual environment in a field of view (FOV) of a user of the head-mounted XR system, wherein the XR assistant avatar follows the user to different spaces, applications, and surfaces within the virtual environment in the FOV, has a first form and a first pose, and is interactable by the user to access an assistant system, and wherein the first form and the first pose are customized to a first context of the user with respect to the virtual environment;
detecting a change from the first context of the user with respect to the virtual environment to a second context of the user with respect to the virtual environment; and
based on the detected change from the first context of the user with respect to the virtual environment to the second context of the user with respect to the virtual environment, rendering, for the display of the head-mounted XR system, a second output image of the XR assistant avatar, wherein the XR assistant avatar is morphed from the first form and the first pose to a second form and a second pose, wherein the second form and the second pose are customized to the second context of the user with respect to the virtual environment,
wherein the first pose and the second pose of the XR assistant avatar each comprise at least one of a position and a movement within the virtual environment.

2. The method of claim 1 , wherein the head-mounted XR system comprises an XR display device, and wherein the display is associated with the XR display device.

3. The method of claim 1 , wherein the head-mounted XR system comprises the assistant system.

4. The method of claim 1 , wherein the virtual environment comprises a real-world environment, and wherein the XR assistant avatar is an augmented-reality (AR) rendering of virtual objects overlaid on the real-world environment.

5. The method of claim 1 , wherein the virtual environment is a virtual-reality (VR) environment, and wherein the XR assistant avatar is a VR rendering.

6. The method of claim 1 , wherein the first or second form of the XR assistant avatar comprises at least one of its voice, speech, emotion, tone, pitch, appearance, size, shape, clothing, orientation, depth, gesture, facial expression, color, shading, outline, brightness, luminescence, and transparency.

7. The method of claim 1 , wherein the detected change from the first context of the user with respect to the virtual environment to the second context of the user with respect to the virtual environment comprises one of a change in an application state associated with a first application on the head-mounted XR system, a change from the first application to a second application on the head-mounted XR system, and a change in the virtual environment.

8. The method of claim 7 , further comprising, responsive to detecting the change from the first context of the user with respect to the virtual environment to the second context of the user with respect to the virtual environment:
registering a first set of actions with the XR assistant avatar, wherein each action in the first set of actions corresponds to a first voice-initiated action executable via the assistant system; and unregistering a second set of actions with the XR assistant avatar, wherein each action in the second set of actions corresponds to a second voice-initiated action executable via the assistant system.

9. The method of claim 1 , further comprising:
determining the second form and the second pose based on a personalization model, wherein the personalization model is a machine-learning model trained on a plurality of prior reactions associated with the user with respect to a plurality of morphing of the XR assistant avatar.

10. The method of claim 1 , further comprising:
detecting one or more user behaviors associated with the user; and
determining, based on the one or more user behaviors, one or more user sentiments associated with the user; and
determining the second form and the second pose based on the one or more user sentiments.

11. The method of claim 1 , wherein rendering the first or second output image of the XR assistant avatar comprises rendering the XR assistant avatar as a human-like avatar.

12. The method of claim 1 , wherein rendering the first or second output image of the XR assistant avatar comprises rendering the XR assistant avatar as an animated object or icon.

13. The method of claim 1 , further comprising:
determining one or more natural-language responses for the XR assistant avatar associated with the rendering of the second output image of the XR assistant avatar, wherein the one or more natural-language responses are customized for the user.

14. The method of claim 13 , wherein the one or more natural-language responses are responsive to a user request from the user.

15. The method of claim 13 , wherein the one or more natural-language responses are customized further based on an application the user is currently engaged with.

16. The method of claim 1 , wherein the first form and the second form are a same form.

17. The method of claim 1 , wherein the first pose and the second pose are a same pose.

18. The method of claim 1 , wherein rendering the first output image of the XR assistant avatar comprises morphing the XR assistant avatar in the first form and the first pose in the field of view of the user in consideration of a first activity of the user associated with the first context of the user, and
wherein rendering the second output image of the XR assistant avatar comprises morphing the XR assistant avatar in the second form and the second pose in the field of view of the user in consideration of a second activity of the user associated with the second context of the user.

19. One or more computer-readable non-transitory non-volatile storage media embodying software that is operable when executed to:
render, for a display of a head-mounted extended-reality (XR) system, a first output image of an XR assistant avatar within a virtual environment in a field of view (FOV) of a user of the head-mounted XR system, wherein the XR assistant avatar follows the user to different spaces, applications, and surfaces within the virtual environment in the FOV, has a first form and a first pose, and is interactable by the user to access an assistant system, and wherein the first form and the first pose are customized to a first context of the user with respect to the virtual environment;
detect a change from the first context of the user with respect to the virtual environment to a second context of the user with respect to the virtual environment; and
based on the detected change from the first context of the user with respect to the virtual environment to the second context of the user with respect to the virtual environment, render, for the display of the head-mounted XR system, a second output image of the XR assistant avatar, wherein the XR assistant avatar is morphed from the first form and the first pose to a second form and a second pose, wherein the second form and the second pose are customized to the second context of the user with respect to the virtual environment,
wherein the first pose and the second pose of the XR assistant avatar each comprise at least one of a position and a movement within the virtual environment.

20. A system comprising: one or more processors; and a non-transitory no

Assignees

• Meta Platforms Inc

Inventors

• Khorshid Mokhtar Mohamed

Classifications

• G06F3/012

  Head tracking input arrangements · CPC title

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06F9/453 Primary

  Help systems · CPC title

• G06T2210/44

  Morphing · CPC title

• G06T19/00

  Manipulating three-dimensional [3D] models or images for computer graphics · CPC title