Rendering XR avatars based on acoustical features

Patent metadata

Field	Value
Publication number	US-12592017-B2
Application number	US-202318353693-A
Country	US
Kind code	B2
Filing date	Jul 17, 2023
Priority date	Jul 17, 2023
Publication date	Mar 31, 2026
Grant date	Mar 31, 2026

Abstract

In one embodiment, a method includes receiving a voice input having acoustic features from a first client system associated with a first user, determining emotions associated with the voice input based on one or more of the acoustic features by machine-learning models, determining facial features for a first extended-reality (XR) avatar representing the first user based on the emotions, and sending instructions for rendering the first XR avatar representing the first user to a second client system associated with a second user, wherein the first XR avatar is rendered with the determined facial features.

First claim

What is claimed is:

1 . A method comprising:
receiving, from a first client system associated with a first user, a voice input having a plurality of acoustic features; determining, based on one or more of the plurality of acoustic features by one or more machine-learning models, one or more emotions associated with the voice input; determining, based on the one or more emotions, one or more facial features for a first extended-reality (XR) avatar representing the first user; determining whether a latency time for rendering the first XR avatar is greater than a threshold time; delaying a transmission of one or more audio signals corresponding to the voice input to a second client system by a delay time, wherein the delay time is determined based on the latency time; and sending, to the second client system associated with a second user, instructions for rendering the first XR avatar representing the first user, wherein the first XR avatar is rendered with the determined one or more facial features.

2 . The method of claim 1 , further comprising:
receiving a visual input corresponding to the voice input, wherein the visual input has a plurality of visual signals, and wherein determining the one or more emotions is further based on one or more of the visual signals.

3 . The method of claim 1 , further comprising:
receiving a motion input corresponding to the voice input, wherein the motion input comprises a plurality of motion signals, and wherein determining the one or more emotions is further based on one or more of the motion signals.

4 . The method of claim 1 , wherein the first user is in a conversation with the second user in an XR environment comprising the first XR avatar.

5 . The method of claim 1 , wherein the one or more machine-learning models comprise an acoustic model.

6 . The method of claim 5 , further comprising:
transforming, by the acoustic model, the voice input to a Mel-scale spectrogram, wherein determining the one or more emotions is further based on the Mel-scale spectrogram.

7 . The method of claim 1 , wherein each of the one or more emotions is associated with one or more of a range, an intensity, or a confidence score.

8 . The method of claim 1 , wherein the one or more emotions are associated with one or more confidence scores, respectively, and wherein the method further comprises:
ranking the one or more emotions based on their confidence scores, wherein determining the one or more facial features is based on a top-ranked emotion.

9 . The method of claim 1 , wherein the one or more emotions are associated with one or more confidence scores, respectively, and wherein the method further comprises:
ranking the one or more emotions based on their confidence scores; and selecting two or more of the ranked emotions based on their rankings; wherein determining the one or more facial features is based on the selected two or more emotions.

10 . The method of claim 9 , further comprising:
generating a blended emotion based on the selected two or more emotions, wherein determining the one or more facial features is based on the blended emotion.

11 . The method of claim 1 , wherein determining the one or more facial features comprises:
identifying a plurality of facial landmarks of the first XR avatar; and mapping the one or more emotions to one or more pre-defined expressive shapes for a face of the first XR avatar based on the plurality of facial landmarks.

12 . The method of claim 1 , further comprising:
generating one or more lip movements using a lip-sync model based on the determined one or more emotions, wherein the first XR avatar is rendered further based on the one or more lip movements.

13 . The method of claim 1 , further comprising:
generating a recommendation of a communication content for the first user based on the determined one or more emotions; and sending, to the first client system, instructions for presenting the recommendation of the communication content.

14 . The method of claim 1 , further comprising:
sending, to an assistant system executing on the first client system, the determined one or more emotions; and receiving, from the assistant system, execution results of one or more tasks determined based on the one or more emotions.

15 . The method of claim 1 , further comprising:
generating a transcription for the voice input, wherein determining the one or more emotions is further based on an analysis of the transcription.

16 . The method of claim 1 , further comprising:
accessing a prior emotion state from a dialog state associated with the first user, wherein determining the one or more emotions is further based on the prior emotion state.

17 . The method of claim 16 , wherein the prior emotion state is determined based on one or more of a prior user input from the first user or a prior user input from the second user.

18 . One or more computer-readable non-transitory storage media embodying software that is operable when executed to:
receive, from a first client system associated with a first user, a voice input having a plurality of acoustic features; determine, based on one or more of the plurality of acoustic features by one or more machine-learning models, one or more emotions associated with the voice input; determine, based on the one or more emotions, one or more facial features for a first extended-reality (XR) avatar representing the first user; determine whether a latency time for rendering the first XR avatar is greater than a threshold time; delay a transmission of one or more audio signals corresponding to the voice input to a second client system by a delay time, wherein the delay time is determined based on the latency time; and send, to the second client system associated with a second user, instructions for rendering the first XR avatar representing the first user, wherein the first XR avatar is rendered with the determined one or more facial features.

19 . A system comprising: one or more processors; and a non-transitory memory coupled to the processors comprising instructions executable by the processors, the processors operable when executing the instructions to:
receive, from a first client system associated with a first user, a voice input having a plurality of acoustic features; determine, based on one or more of the plurality of acoustic features by one or more machine-learning models, one or more emotions associated with the voice input; determine, based on the one or more emotions, one or more facial features for a first extended-reality (XR) avatar representing the first user; determine whether a latency time for rendering the first XR avatar is greater than a threshold time; delay a transmission of one or more audio signals corresponding to the voice input to a second client system by a delay time, wherein the delay time is determined based on the latency time; and send, to the second client system associated with a second user, instructions for rendering the first XR avatar representing the first user, wherein the first XR avatar is rendered with the determined one or more facial features.

20 . The system of claim 19 , wherein the non-transitory memory further comprises instructions executable by the processors to:
receive a visual input corresponding to the voice input, wherein the visual input has a plurality of visual signals, and wherein determining the one or more emotions is further based on one or more of the visual signals.

Assignees

• Meta Platforms Inc

Inventors

• Harpale Abhay Suresh
• Ediz Meryem Pinar Donmez
• Malik Kshitiz
• Muzaffar Omer
• Gupta Mridul
• Janakiraman Vijay Manikandan

Classifications

• G10L25/63 Primary

  for estimating an emotional state · CPC title

• G10L25/18

  the extracted parameters being spectral information of each sub-band · CPC title

• G06T13/40

  of characters, e.g. humans, animals or virtual beings · CPC title

• G06T13/205 Primary

  driven by audio data · CPC title

• G06V40/174

  Facial expression recognition · CPC title