Generating instructions for nonverbal movements of a virtual character

Patent metadata

Field	Value
Publication number	US-9721373-B2
Application number	US-201414209473-A
Country	US
Kind code	B2
Filing date	Mar 13, 2014
Priority date	Mar 14, 2013
Publication date	Aug 1, 2017
Grant date	Aug 1, 2017

Abstract

Programs for creating a set of behaviors for lip sync movements and nonverbal communication may include analyzing a character's speaking behavior through the use of acoustic, syntactic, semantic, pragmatic, and rhetorical analyses of the utterance. For example, a non-transitory, tangible, computer-readable storage medium may contain a program of instructions that cause a computer system running the program of instructions to: receive a text specifying words to be spoken by a virtual character; extract metaphoric elements, discourse elements, or both from the text; generate one or more mental state indicators based on the metaphoric elements, the discourse elements, or both; map each of the one or more mental state indicators to a behavior that the virtual character should display with nonverbal movements that convey the mental state indicators; and generate a set of instructions for the nonverbal movements based on the behaviors.

First claim

The invention claimed is:

1. A non-transitory, tangible, computer-readable storage medium containing a program of instructions that cause a computer system running the program of instructions to:
receive a text specifying words to be spoken by a virtual character;
extract metaphoric elements, discourse elements, or both from the text;
generate one or more mental state indicators based on the metaphoric elements, the discourse elements, or both;
map each of the one or more mental state indicators to a behavior that the virtual character should display with nonverbal movements that convey the mental state indicators; and
generate a set of instructions for the nonverbal movements based on the behaviors,
wherein the program of instructions also causes the computer system running the program of instructions to do one of a, b, or c:
a. extract reference elements, valence elements, or both from the text; and wherein generate the one or more mental state indicators is further based on the reference elements, valence elements, or both,
b. receive an audio file corresponding to the text; extract emphasis moments, agitation moments, speech disfluency, or a combination thereof from the audio file; and generate the one or more mental state indicators further based on the emphasis moments, the agitation moments, the speech disfluency, or the combination thereof, or
c. generate a set of instructions for a listener's nonverbal movements in response to the set of instructions for nonverbal movements of the virtual character.

2. The medium of claim 1 , wherein the program of instructions causes the computer system running the program of instructions to do the subpart labeled a.

3. The medium of claim 1 , wherein the program of instructions causes the computer system running the program of instructions to do the subpart labeled b.

4. The medium of claim 1 , wherein generate the set of instructions for the nonverbal movements involves elimination of some of the behaviors.

5. The medium of claim 1 , wherein the program of instructions also cause the computer system running the program of instructions to: convert an utterance into the text and the audio file.

6. The medium of claim 5 , wherein the utterance is a real-time input.

7. The medium of claim 1 , wherein the set of instructions for nonverbal movements of the virtual character based on a gender or a personality of the virtual character.

8. The medium of claim 1 , wherein the program of instructions that also cause the computer system running the program of instructions to: produce a set of instructions for lip sync movements of the virtual character based on the text and the audio file.

9. The medium of claim 8 , wherein the program of instructions that also cause the computer system running the program of instructions to: produce a set of instructions for animation of the virtual character based on the set of instructions for nonverbal movements of the virtual character and the set of instructions for lip sync movements of the virtual character.

10. The medium of claim 1 , wherein the program of instructions also cause the computer system running the program of instructions to: generate a set of instructions for a listener's nonverbal movements in response to the set of instructions for nonverbal movements of the virtual character.

11. The medium of claim 1 , wherein the behaviors include at least one of: head movements, gazing, blinking, saccadic eye movements, hand gesturing, body gesturing, or facial expressions.

12. A non-transitory, tangible, computer-readable storage medium containing a program of instructions that cause a computer system running the program of instructions to:
receive a text specifying words to be spoken by a virtual character;
receive an audio file corresponding to the text;
extract metaphoric elements, discourse elements, reference elements, valence elements, or a combination thereof from the text;
extract agitation moments, speech disfluency, or both from the audio file;
generate one or more mental state indicators based on the metaphoric elements, the discourse elements, the reference elements, the valence elements, or the combination thereof and the agitation moments, the speech disfluency, or both;
map each of the one or more mental state indicators to a behavior that the virtual character should display with nonverbal movements that convey the mental state indicators; and
generate a set of instructions for the nonverbal movements based on the behaviors.

13. The medium of claim 12 , wherein the program of instructions also cause the computer system running the program of instructions to: extract emphasis moments, or both from the audio file; and generate the one or more mental state indicators further based on the emphasis moments, or both.

14. The medium of claim 12 , wherein the program of instructions also cause the computer system running the program of instructions to: convert an utterance into the text and the audio file.

15. The medium of claim 12 , wherein the program of instructions that also cause the computer system running the program of instructions to: produce a set of instructions for lip sync movements of the virtual character based on the text and the audio file.

16. A non-transitory, tangible, computer-readable storage medium containing a program of instructions that cause a computer system running the program of instructions to:
receive a text specifying words to be spoken by a virtual character;
extract metaphoric elements, discourse elements, or both from the text;
generate one or more mental state indicators that the virtual character should display with nonverbal movements based on the metaphoric elements, the discourse elements, or both;
map each of the one or more mental state indicators to a set of possible behaviors that the virtual character should display with nonverbal movements that convey the mental state indicators; and
generate a set of instructions for the nonverbal movements based on the behaviors,
wherein the program of instructions also causes the computer system running the program of instructions to do one of a or b:
a. extract reference elements, valence elements, or both from the text; and generate the one or more mental state indicators further based on the reference elements, valence elements, or both, or
b. receive an audio file corresponding to the text; extract emphasis moments, agitation moments, speech disfluency, or a combination thereof from the audio file; and generate the one or more mental state indicators further based on the emphasis moments, the agitation moments, the speech disfluency, or the combination thereof.

17. The medium of claim 16 , wherein the program of instructions cause the computer system running the program of instructions to do the subpart labeled a.

18. The medium of claim 16 , wherein the program of instructions causes the computer system running the program of instructions to do the subpart labeled b.

19. The medium of claim 16 , wherein the program of instructions also cause the computer system running the program of instructions to: convert an utterance into the text and the audio file.

20. The medium of claim 16 , wherein the program of instructions that also cause the computer system running the program of instructions to: produce a set of instructions for lip sync movements of the virtual character based on the text and the audio file.

Assignees

• Marsella Stacy
• Univ Southern California

Inventors

• Marsella Stacy

Classifications

• G06T13/00

  Animation · CPC title

• G06T13/40 Primary

  of characters, e.g. humans, animals or virtual beings · CPC title

• G06T13/205

  driven by audio data · CPC title