Evaluating natural language processing components

Patent metadata

Field	Value
Publication number	US-12197871-B2
Application number	US-202217973674-A
Country	US
Kind code	B2
Filing date	Oct 26, 2022
Priority date	Aug 26, 2020
Publication date	Jan 14, 2025
Grant date	Jan 14, 2025

Abstract

Techniques for evaluating a natural language understanding (NLU) component and determining an action to resolve an issue processing a user input are described. The system determines which component is invoked by a baseline NLU component is processing the user input, and which component is invoked by an updated NLU component. Based on that information, the system selects the action to resolve the updated NLU component generating an undesired response to the user input.

First claim

What is claimed is:

1. A computer-implemented method, comprising:
receiving first data corresponding to processing of a first user input using a first component, the first data including first output data;
receiving second data corresponding to processing of the first user input using a first updated component, the second data including second output data different from the first output data;
determining the second data results in an undesired response; and
determining a second updated component using the first updated component, the determining of the second updated component being based at least in part on processing of the first data with respect to the second data.

2. The computer-implemented method of claim 1 , further comprising:
determining the second output data corresponds to a message indicating an error in processing the first user input.

3. The computer-implemented method of claim 1 , further comprising:
determining that processing of the first user input by the second updated component results in a desired response; and
storing the second updated component for processing of future user inputs.

4. The computer-implemented method of claim 1 , wherein:
the first component comprises a first model and a second model; and
the first updated component comprises the first model and a third model, the third model corresponding to an updated version of the second model.

5. The computer-implemented method of claim 4 , further comprising:
determining the undesired response corresponds to processing by the third model.

6. The computer-implemented method of claim 5 , further comprising:
determining that first grammar data corresponding to the second model is different than second grammar data corresponding to the third model; and
determining third grammar data by updating the second grammar data.

7. The computer-implemented method of claim 1 , wherein:
the first component comprises a first model and a second model;
the first updated component comprises a third model and a fourth model; and
the second updated component comprises the third model and a fifth model, the fifth model corresponding to an updated version of the fourth model.

8. The computer-implemented method of claim 1 , wherein;
the first component comprises a first model and a second model;
the first updated component comprises a third model; and
the method further comprises:
determining that the first data indicates that the first model was invoked during processing of the first user input,
determining that the second data indicates that the third model was invoked during processing of the first user input,
processing the first output data with respect to the second output data,
based at least in part on the processing of the first output data with respect to the second output data, determining training data to include a set of user inputs corresponding to the first user input, the set of user inputs associated with a positive label, and
training, using the training data, the first updated component to determine the second updated component.

9. The computer-implemented method of claim 8 , further comprising:
receiving third data representing processing of the first user input using the second updated component, the second updated component comprising a third model and a fourth model, the third data comprising third output data;
determining that processing of the first user input by the second updated component results in the undesired response;
determining that the third data indicates that the third model was invoked during processing of the first user input;
processing the first output data with respect to the third output data; and
generating a fifth model corresponding to the first user input, the fifth model to be included in the second updated component.

10. The computer-implemented method of claim 1 , wherein:
the first component comprises a first model and a second model;
the first updated component comprises a third model; and
the method further comprises:
determining that the first data indicates that the first model was invoked during processing of the first user input,
determining that the second data indicates that the third model was invoked during processing of the first user input,
processing the first output data with respect to the second output data,
based at least in part on the processing of the first output data with respect to the second output data, determining training data to include a set of user inputs corresponding to the first user input, the set of user inputs associated with a negative label, and
training, using the training data, the first updated component to determine the second updated component.

11. A system comprising:
at least one processor; and
at least one memory comprising instructions that, when executed by the at least one processor, cause the system to:
receive first data corresponding to processing of a first user input using a first component, the first data including first output data;
receive second data corresponding to processing of the first user input using a first updated component, the second data including second output data different from the first output data;
determine the second data results in an undesired response; and
determine a second updated component using the first updated component, wherein determination of the second updated component being based at least in part on processing of the first data with respect to the second data.

12. The system of claim 11 , wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to:
determine the second output data corresponds to a message indicating an error in processing the first user input.

13. The system of claim 11 , wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to:
determine that processing of the first user input by the second updated component results in a desired response; and
store the second updated component for processing of future user inputs.

14. The system of claim 11 , wherein:
the first component comprises a first model and a second model; and
the first updated component comprises the first model and a third model, the third model corresponding to an updated version of the second model.

15. The system of claim 14 , wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to:
determine the undesired response corresponds to processing by the third model.

16. The system of claim 15 , wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to:
determine that first grammar data corresponding to the second model is different than second grammar data corresponding to the third model; and
determine third grammar data by updating the second grammar data.

17. The system of claim 11 , wherein:
the first component comprises a first model and a second model;
the first updated component comprises a third model and a fourth model; and
the second updated component comprises the third model and a fifth model, the fifth model corresponding to an updated version of the fourth model.

18. The system of claim 11 , wherein:
the first component comprises a first model and a second model;
the first updated component comprises a third model; and
the at least one memory

Assignees

• Amazon Tech Inc

Inventors

• Bhowmik Pavel
• Gens Melanie C B
• Midha Sachin
• Gupta Rahul
• Venkatapathy Sriram
• Zhang Xinhong
• Kumar Anoop
• Sonawane Pooja Sanjay
• Ingbar Samuel Harry

Classifications

• G10L2015/223

  Execution procedure of a spoken command · CPC title

• G10L15/1822

  Parsing for meaning understanding · CPC title

• G06F40/295

  Named entity recognition · CPC title

• G06F40/289

  Phrasal analysis, e.g. finite state techniques or chunking · CPC title

• G06F40/30 Primary

  Semantic analysis · CPC title