Methods and apparatus for conversation coach

Patent metadata

Field	Value
Publication number	US-9691296-B2
Application number	US-201414294326-A
Country	US
Kind code	B2
Filing date	Jun 3, 2014
Priority date	Jun 3, 2013
Publication date	Jun 27, 2017
Grant date	Jun 27, 2017

Abstract

In exemplary implementations of this invention, a display screen and speakers present an audiovisual display of an animated character to a human user during a conversational period of a coaching session. The virtual character asks questions, listens to the user, and engages in mirroring and backchanneling. A camera and microphone gather audiovisual data regarding behavior of the user. After the conversational period, the display screen and speakers display feedback to the user regarding the user's behavior. For example, the feedback may include a plot of the user's smiles over time, or information regarding prosody of the user's speech. The feedback may also include playing a video of the user that was recorded during the conversational period. The feedback may also include a timeline of the human user's behavior. The virtual coaching may be provided over the Internet.

First claim

What is claimed is:

1. A method of interacting with a human user, which interacting is performed by a display screen, a speaker, a video camera and a microphone and which interacting comprises:
(a) the video camera and the microphone gathering audiovisual data regarding the user during a period of time;
(b) the display screen and the speaker presenting to the user an audiovisual display of an animated character, such that the animated character
(i) engages in a verbal conversation with the user during the period, and
(ii) mirrors, during the conversation (A) posture of the user that occurs during the conversation and (B) other actions of the user that occur during the conversation; and

c) the display screen and the speaker providing, after the period, visual and audio feedback to the user regarding behavior of the user during the period, which feedback includes
(i) visual information in a text, chart or graph format, which visual information includes information regarding smiles of the user and prosody of the user's speech during the period, and
(ii) a display of an audiovisual recording of the behavior.

2. The method of claim 1 , wherein the visual information regarding smiles is a plot of smile intensity versus time.

3. The method of claim 1 , wherein the animated character appears, at least once during the period, to mirror (i) head orientation of the user or (ii) smiles of the user.

4. The method of claim 1 , wherein the animated character appears during the period to ask a question to the user at a time when the user is not speaking.

5. The method of claim 1 , wherein:
(a) during a majority of the period, the animated character maintains a neutral expression;
(b) when a human user smiles during the period, the animated character responds with a smile;
(c) at times during the period, the animated character asks a question, appears to listen to an answer given by the user in response to the question, provides a verbal acknowledgement after the answer, and then asks another question; and
(d) at times during the period, the animated character engages in head nods.

6. The method of claim 1 , wherein the visual information in text, chart or graph format includes information that compares a behavioral feature of the behavior of the user to a statistical measure of the behavioral feature as exhibited by other humans.

7. The method of claim 1 , wherein the visual information in text, chart or graph format includes information about any one or more of the following: (a) rate of speech; (b) volume of speech; (c) pauses in speech; (d) pitch of speech, or (e) a pattern of variation of one or of speech, which one or more features of speech include one or more of rate of speech, volume of speech, pauses in speech, or pitch of speech.

8. The method of claim 1 , wherein the visual information in a text, chart or graph format includes information regarding head nods by the user.

9. The method of claim 1 , wherein the visual information in text, chart or graph format includes information that specifies occurrence, number, percentage or frequency of, or that comprises a text transcription of, sounds or words uttered by the user that are in a specific class of sounds or words, which specific class of sounds or words does not include all of the sounds and words uttered by the user during the period.

10. The method of claim 1 , wherein the animated character appears to function as an interviewer in a mock job interview during the period.

11. The method of claim 1 , wherein one or processors determine relevance of content of the user's speech.

12. A method comprising, in combination:
(a) a set of one or more servers accepting sensor data, which sensor data has been transmitted through a network and includes audiovisual data gathered by a microphone and a video camera regarding a first human user during a period of time;
(b) a computer analyzing-the sensor data; and
(c) at least one server, out of the set, outputting signals for transmission over the network, which signals encode instructions for controlling a display screen and a speaker to interact with the user
(i) by displaying an animated character to the user, such that the animated character
(A) engages in a verbal conversation with the user during the period, and
(B) mirrors, during the conversation (1) posture of the user that occurs during the conversation and (2) other actions of the user that occur during the conversation, and

(ii) by displaying feedback to the first human user regarding behavior of the first human user that occurs during the period, which feedback includes
(A) visual information in a text, chart or graph format, which visual information includes information regarding smiles or prosody of the first human user during the period, and
(B) a display of an audiovisual recording of the behavior.

13. The method of claim 12 , wherein the one or more servers: (a) output signals to share, with other computers that are nodes in a social network, information regarding the behavior of the first human user; and (b) accept signals that are indicative of feedback from other humans regarding the behavior of the first human user.

14. The method of claim 12 , wherein the animated character appears, at multiple times during the period, to engage in actions that are responsive to the first human user, including (i) asking questions when the first human user is not speaking, and (ii) nodding or moving the animated character's head, such that the animated character displays all, less than all, or none of the responsive actions at any given time during the period.

15. The method of claim 12 , wherein:
(a) for each respective user, out of multiple human users, the audiovisual data captures behavior of the respective user during a period in which an animated character is displayed to the respective user;
(b) the method further comprises using a computer to calculate a statistical measure regarding an aspect of behavior of a set of users, out of the multiple human users.

16. The method of claim 15 , wherein the animated character interacts with the user by:
(a) during a majority of the period, maintaining a neutral expression;
(b) when a human user smiles during the period, responding with a smile;
(c) at times during the period, asking a question, appearing to listen to an answer given by the user in response to the question, providing a verbal acknowledgement after the answer, and then asking another question; and
(d) at times during the period, engaging in head nods.

17. The method of claim 12 , wherein the visual information in a text, chart or graph format includes information that compares a behavioral feature of the behavior of the first human user to a statistical measure of the behavioral feature as exhibited by other humans.

18. The method of claim 17 , wherein the first human user and each of the other humans are members of a class, which class is defined, at least in part, by one or more of age, gender, occupation, ethnicity or education level of the members.

19. The method of claim 18 , wherein the animated character appears, at times during the period, to interact with the user by mirroring (i) head orientation of the user or (ii) smiles of the user.

Assignees

• Hoque Mohammed Ehasanul
• Picard Rosalind
• Massachusetts Inst Technology

Inventors

• Hoque Mohammed Ehasanul
• Picard Rosalind

Classifications

• G09B5/065

  Combinations of audio and video presentations, e.g. videotapes, videodiscs, television systems · CPC title

• G09B7/00

  Electrically-operated teaching apparatus or devices working with questions and answers (mechanically operated G09B3/00; computing arrangements G06F) · CPC title

• G09B19/04 Primary

  Speaking (with audible presentation of the material to be studied G09B5/04) · CPC title