Automatic updating of confidence scoring functionality for speech recognition systems with respect to a receiver operating characteristic curve

Patent metadata

Field	Value
Publication number	US-9330665-B2
Application number	US-201113977174-A
Country	US
Kind code	B2
Filing date	Jan 7, 2011
Priority date	Jan 7, 2011
Publication date	May 3, 2016
Grant date	May 3, 2016

Abstract

Automatically adjusting confidence scoring functionality is described for a speech recognition engine. Operation of the speech recognition system is revised so as to change an associated receiver operating characteristic (ROC) curve describing performance of the speech recognition system with respect to rates of false acceptance (FA) versus correct acceptance (CA). Then a confidence scoring functionality related to recognition reliability for a given input utterance is automatically adjusted such that where the ROC curve is better for a given operating point after revising the operation of the speech recognition system, the adjusting reflects a double gain constraint to maintain FA and CA rates at least as good as before revising operation of the speech recognition system.

First claim

What is claimed is:

1. A method comprising:
determining an original receiver operating characteristic (ROC) curve describing performance of a speech recognition system with respect to an original rate of false acceptance (FA) of the speech recognition system versus an original rate of correct acceptance (CA) of the speech recognition system;
changing an algorithm used by the speech recognition system for sentence confidence scores, wherein changing the algorithm results in a new ROC curve with respect to a new rate of FA of the speech recognition system versus a new rate of CA of the speech recognition system;
receiving a user specification of relative importance of the new rate of FA versus the new rate of CA; and
based on the relative importance of the new rate of FA versus the new rate of CA, adjusting a confidence scoring functionality related to recognition reliability for a given input utterance,
wherein at or above a given operating point of the speech recognition system, the new ROC curve reflects a double gain constraint relative to the original ROC curve, such that the new rate of FA is equal to or less than the original rate of FA, and the new rate of CA is equal to or greater than the original rate of CA.

2. The method of claim 1 , wherein below the given operating point of the speech recognition system, the new ROC curve minimizes worsening of the rate of FA and the rate of CA.

3. The method of claim 1 , comprising:
one or more original settings of the confidence scoring functionality to corresponding one or more new settings of the confidence scoring functionality.

4. The method of claim 3 , comprising:
estimating the mapping using representative labeled training data.

5. The method of claim 4 , wherein estimating the mapping using representative labeled training data comprises estimating a first mapping function that maps confidence scores for the original ROC curve to first posterior probabilities, and estimating a second mapping function that maps confidence scores for the new ROC curve to second posterior probabilities.

6. The method of claim 5 , comprising:
determining an equivalence relationship between the first mapping function and the second mapping function,
wherein adjusting the confidence scoring functionality comprises adjusting the confidence scoring functionality based on the equivalence relationship between the first mapping function and the second mapping function.

7. The method of claim 6 , comprising:
determining a range of discrete integer confidence values for the equivalence relationship; and
performing linear interpolation to determine any missing values in the range of discrete integer confidence values for the equivalence relationship.

8. The method of claim 1 , wherein the confidence scoring functionality includes confidence score thresholds that define for a given set of circumstances whether to accept, reject, or confirm a given input utterance.

9. The method of claim 8 , wherein adjusting the confidence scoring functionality includes updating the confidence score thresholds.

10. The method of claim 9 , wherein the user specification of relative importance of the new rate of FA versus the new rate of CA comprises input establishing a priority favoring the rate of FA or the rate of CA.

11. The method of claim 9 , wherein the confidence score thresholds on the new ROC curve are determined based on accepting and rejecting an equal percentage of input utterances as the confidence score thresholds on the original ROC curve.

12. The method of claim 1 , wherein receiving the user specification of relative importance of the new rate of FA versus the new rate of CA comprises receiving the user specification of relative importance via an adjustable slider of a graphical user interface of the speech recognition system, wherein a position of the adjustable slider corresponds to the relative importance of the new rate of FA versus the new rate of CA.

13. The method of claim 1 ,
wherein the relative importance of the new rate of CA is greater than the relative importance of the new rate of FA, and
wherein based on the relative importance of the new rate of CA being greater than the relative importance of the new rate of FA, adjusting the confidence scoring functionality related to the recognition reliability for the given input utterance comprises maximizing the new rate of CA.

14. The method of claim 1 ,
wherein the relative importance of the new rate of FA is greater than the relative importance of the new rate of CA, and
wherein based on the relative importance of the new rate of FA being greater than the relative importance of the new rate of CA, adjusting the confidence scoring functionality related to the recognition reliability for the given input utterance comprises minimizing the new rate of FA.

15. The method of claim 1 , wherein a search space used by the speech recognition system for the given input utterance is reduced to a region that satisfies the double gain constraint.

16. The method of claim 1 , comprising:
updating an acoustic model used by the speech recognition system, wherein the updated acoustic model is associated with the new ROC curve with respect to the new rate of FA of the speech recognition system versus the new rate of CA of the speech recognition system.

17. Non-transitory computer-readable media storing executable instructions that, when executed by one or more processors, cause a speech recognition system to:
determine an original receiver operating characteristic (ROC) curve describing performance of the speech recognition system with respect to an original rate of false acceptance (FA) of the speech recognition system versus an original rate of correct acceptance (CA) of the speech recognition system;
change an algorithm used by the speech recognition system for sentence confidence scores, wherein changing the algorithm results in a new ROC curve with respect to a new rate of FA of the speech recognition system versus a new rate of CA of the speech recognition system;
receive a user specification of relative importance of the new rate of FA versus the new rate of CA; and
based on the relative importance of the new rate of FA versus the new rate of CA, adjust a confidence scoring functionality related to recognition reliability for a given input utterance,
wherein at or above a given operating point of the speech recognition system, the new ROC curve reflects a double gain constraint relative to the original ROC curve, such that the new rate of FA is equal to or less than the original rate of FA, and the new rate of CA is equal to or greater than the original rate of CA.

18. The non-transitory computer-readable media of claim 17 , wherein below the given operating point of the speech recognition system, the new ROC curve minimizes worsening of the rate of FA and the rate of CA.

19. The non-transitory computer-readable media of claim 17 , wherein the executable instructions, when executed by the one or more processors, cause the speech recognition system to:
map one or more original settings of the confidence scoring functionality to corresponding one or more new settings of the confidence scoring functionality.

20. The non-transitory computer-readable media of claim 17 , wherein the confidence scoring functionality includes confidence score thresholds that define for a given set of circumstances whether to accept, reject, or confirm a given input utterance.

21. The non-transitory computer-readable media of cl

Assignees

• Morales Nicolas
• Connolly Dermot
• Halberstadt Andrew
• Nuance Communications Inc

Inventors

• Morales Nicolas
• Connolly Dermot
• Halberstadt Andrew

Classifications

• G10L15/22 Primary

  Procedures used during a speech recognition process, e.g. man-machine dialogue · CPC title

• G10L2015/0636

  Threshold criteria for the updating · CPC title

• G10L15/01

  Assessment or evaluation of speech recognition systems · CPC title

• G10L15/065 Primary

  Adaptation · CPC title

• G10L2015/0635

  updating or merging of old and new templates; Mean values; Weighting · CPC title