System and method for using prosody for voice-enabled search

Patent metadata

Field	Value
Publication number	US-10002608-B2
Application number	US-88495910-A
Country	US
Kind code	B2
Filing date	Sep 17, 2010
Priority date	Sep 17, 2010
Publication date	Jun 19, 2018
Grant date	Jun 19, 2018

Abstract

Disclosed herein are systems, methods, and non-transitory computer-readable storage media for approximating relevant responses to a user query with voice-enabled search. A system practicing the method receives a word lattice generated by an automatic speech recognizer based on a user speech and a prosodic analysis of the user speech, generates a reweighted word lattice based on the word lattice and the prosodic analysis, approximates based on the reweighted word lattice one or more relevant responses to the query, and presents to a user the responses to the query. The prosodic analysis examines metalinguistic information of the user speech and can identify the most salient subject matter of the speech, assess how confident a speaker is in the content of his or her speech, and identify the attitude, mood, emotion, sentiment, etc. of the speaker. Other information not described in the content of the speech can also be used.

First claim

We claim:

1. A method comprising:
receiving a word lattice generated by an automatic speech recognizer processing a query, wherein the word lattice is weighted according to the query;
identifying a policy which allows use of a user emotional state in responding to a user who produced the query;
performing, via a processor of the automatic speech recognizer, a prosodic analysis of the query, wherein the prosodic analysis identifies an audible gesture in the query and a rhythm of words spoken in the query;
identifying, according to the prosodic analysis, the user emotional state;
reweighting, via the processor, the word lattice according to the prosodic analysis, the user emotional state and one of a time of day, a time of year, and a behavioral history of the user, to yield a reweighted word lattice;
determining, via the processor and according to the reweighted word lattice, a response to the query, the response addressing the audible gesture; and
presenting to the user the response to the query.

2. The method of claim 1 , wherein the prosodic analysis further examines metalinguistic information of the query according to one or more of a tonality, a volume, a stress, an intonation, and a speed of the query.

3. The method of claim 2 , wherein the prosodic analysis identifies a most salient subject matter of the query.

4. The method of claim 2 , wherein the prosodic analysis assesses a confidence of the user in content of the query.

5. The method of claim 2 , wherein the prosodic analysis identifies a mood of the user.

6. The method of claim 1 , wherein determining the response is performed by composing the reweighted word lattice with a search finite state transducer according to a plurality of pre-indexed documents.

7. The method of claim 6 , further comprising generating a finite state transducer according to composing the reweighted word lattice.

8. The method of claim 1 , wherein the audible gesture comprises one of a laugh, a sob, a cough, and a yawn.

9. A system comprising:
a processor; and
a computer-readable storage medium having instructions stored which, when executed by the processor, cause the processor to perform operations comprising:
receiving a word lattice generated by an automatic speech recognizer a query, wherein the word lattice is weighted according to the query;
identifying a policy which allows use of a user emotional state in responding to a user who produced the query;
performing a prosodic analysis of the query, wherein the prosodic analysis identifies an audible gesture in the query and a rhythm of words spoken in the query;
identifying, according to the prosodic analysis, the user emotional state;
reweighting the word lattice according to the prosodic analysis, the user emotional state and one of a time of day, a time of year, and a behavioral history of the user to yield a reweighted word lattice;
determining, according to the reweighted word lattice, a response to the query, the response addressing the audible gesture; and
presenting to the user the response to the query.

10. The system of claim 9 , wherein the prosodic analysis examines metalinguistic information of the query according to one or more of a tonality, a volume, a stress, an intonation, and a speed of the query.

11. The system of claim 10 , wherein the prosodic analysis identifies a most salient subject matter of the query.

12. The system of claim 10 , wherein the prosodic analysis assesses a confidence of the user in content of the query.

13. The system of claim 10 , wherein the prosodic analysis identifies a mood of the user.

14. A computer-readable storage device having instructions stored which, when executed by a computing device, cause the computing device to perform operations comprising:
receiving a word lattice generated by an automatic speech recognizer processing a query, wherein the word lattice is weighted according to the query;
identifying a policy which allows use of a user emotional state in responding to a user who produced the query;
performing a prosodic analysis of the query, wherein the prosodic analysis identifies an audible gesture in the query and a rhythm of words spoken in the query;
identifying, according to the prosodic analysis, the user emotional state;
reweighting the word lattice according to the prosodic analysis, the user emotional state and one of a time of day, a time of year, and a behavioral history of the user, to yield a reweighted word lattice;
determining, according to the reweighted word lattice, a response to the query, the response addressing the audible gesture; and
presenting to the user the response to the query.

15. The computer-readable storage device of claim 14 , wherein determining the response is performed by composing the reweighted word lattice with a search finite state transducer according to a plurality of pre-indexed documents.

16. The computer-readable storage device of claim 15 , having additional instructions stored which, when executed by the computing device, result in operations comprising generating a finite state transducer according to composing the reweighted word lattice.

17. The computer-readable storage device of claim 16 , having additional instructions stored which, when executed by the computing device, result in operations comprising reranking, according to the finite state transducer, the word lattice generated by the automatic speech recognizer.

Assignees

• Bangalore Srinivas
• Feng Junlan
• Johnston Michael
• Mishra Taniya
• Nuance Communications Inc

Inventors

• Bangalore Srinivas
• Feng Junlan
• Johnston Michael
• Mishra Taniya

Classifications

• G10L25/54

  for retrieval · CPC title

• G10L2015/226

  using non-speech characteristics · CPC title

• G10L25/63

  for estimating an emotional state · CPC title

• G10L2015/227

  of the speaker; Human-factor methodology · CPC title

• G10L15/1807 Primary

  using prosody or stress · CPC title