Cohort determination in natural language processing

Patent metadata

Field	Value
Publication number	US-12112752-B1
Application number	US-202217688279-A
Country	US
Kind code	B1
Filing date	Mar 7, 2022
Priority date	Mar 7, 2022
Publication date	Oct 8, 2024
Grant date	Oct 8, 2024

Abstract

Devices and techniques are generally described for cohort determination in natural language processing. In various examples, a first natural language input to a natural language processing system may be determined. The first natural language input may be associated with a first account identifier. A first machine learning model may determine first data representing one or more words of the first natural language input. A second machine learning model may determine second data representing one or more acoustic characteristics of the first natural language input. Third data may be determined, the third data including a predicted performance for processing the first natural language input by the natural language processing system. The third data may be determined based on the first data representation and the second data representation.

First claim

What is claimed is:

1. A computer-implemented method comprising:
determining a first account identifier associated with a natural language processing system;
receiving a natural language input associated with the first account identifier over a first past time period;
generating, using a language model, first data representing the natural language input;
receiving first audio data representing the natural language input;
generating, using an acoustic model, second data representing the first audio data;
determining third data representing a performance metric of the natural language processing system, the performance metric associated with predicted error during processing of the natural language input;
generating fourth data by concatenating at least the first data, the second data, and the third data;
generating, using an unsupervised clustering algorithm, a plurality of clusters of account identifiers, wherein a first cluster of the plurality of clusters includes the fourth data and a plurality of other data representations;
determining an average score of the performance metric for the first cluster;
determining that the average score of the performance metric is associated with underperformance of the natural language processing system for the natural language input;
generating a training data set for a first machine learning model of the natural language processing system, the training data set including the fourth data and the plurality of other data representations; and
generating updated parameters of the first machine learning model using the training data set.

2. The method of claim 1 , further comprising:
determining first metadata associated with the first account identifier, the first metadata identifying a geographical location associated with the first account identifier, wherein the fourth data is generated using the first metadata, and wherein the plurality of other data representations of the first cluster are associated with the first metadata identifying the geographical location.

3. The method of claim 1 , further comprising:
generating the plurality of clusters using a first homogeneity loss, the first homogeneity loss maximizing a first similarity of the plurality of other data representations with respect to one another; and
generating the plurality of clusters using a second homogeneity loss, the second homogeneity loss maximizing a second similarity of performance metrics associated with the plurality of other data representations.

4. A method comprising:
receiving a first natural language input to a natural language processing system, the first natural language input being associated with a first account identifier;
determining, using a first machine learning model, first data representing one or more words of the first natural language input;
determining, using a second machine learning model, second data representing one or more acoustic characteristics of the first natural language input;
determining, based at least in part on the first data and the second data, third data representing a predicted performance for processing the first natural language input by the natural language processing system; and
determining, based at least in part on the predicted performance, a first cluster associated with the first natural language input, wherein the first cluster comprises data representing past natural language inputs.

5. The method of claim 4 , further comprising:
determining, using an unsupervised machine learning algorithm, a plurality of clusters of account identifiers based at least in part on the first data and the second data, wherein the first cluster is among the plurality of clusters, and wherein the first account identifier is included in the first cluster of the plurality of clusters; and
determining fourth data representing a performance level of the natural language processing system for processing natural language inputs received from account identifiers associated with the first cluster.

6. The method of claim 4 , further comprising:
determining, using an unsupervised machine learning algorithm, a plurality of clusters of account identifiers based at least in part on the first data and the second data, wherein the first cluster is among the plurality of clusters, and wherein the first account identifier is included in the first cluster of the plurality of clusters;
determining, fourth data comprising an aggregated performance level associated with the first cluster; and
including account identifiers of the first cluster in a training data set based at least in part on the fourth data.

7. The method of claim 4 , further comprising:
determining, based at least in part on the third data, a cohort of account identifiers associated with the predicted performance;
generating a training data set comprising feature data representing natural language inputs associated with the account identifiers of the cohort; and
generating at least one updated parameter of a third machine learning model using the training data set, the third machine learning model being associated with the natural language processing system.

8. The method of claim 4 , further comprising:
determining, by a neural network, the predicted performance, wherein the neural network is trained using a first training instance comprising a first feature representation of a second natural language input and label data comprising a performance score of the natural language processing system for processing the second natural language input.

9. The method of claim 4 , further comprising:
determining a third data representation comprising a mel-frequency cepstral coefficient of the first natural language input; and
generating a first combined feature vector associated with the first account identifier based at least in part on a concatenation of a first vector representing the first data, a second vector representing the second data, and a third vector representing the third data.

10. The method of claim 9 , further comprising:
determining the first cluster comprising the first combined feature vector and a second combined feature vector, wherein the second combined feature vector is associated with a second account identifier, and wherein the first combined feature vector and the second combined feature vector are included in the first cluster based at least in part on a similarity metric used to determine a distance between the first combined feature vector and the second combined feature vector in a feature space common to both the first combined feature vector and the second combined feature vector.

11. The method of claim 4 , further comprising:
determining a plurality of machine learning models effective to perform a first natural language processing task;
determining, from among the plurality of machine learning models, a third machine learning model associated with the first cluster; and
selecting the third machine learning model to process the first natural language input based at least in part on the first cluster being associated with the first natural language input.

12. The method of claim 4 , further comprising:
determining, using an unsupervised machine learning algorithm, a plurality of clusters of account identifiers based at least in part on the first data and the second data, wherein the first cluster is among the plurality of clusters, and wherein the first account identifier is included in the first cluster of the plurality of clusters;
determining a first error rate associated with the first cluster of the plurality of clusters;
determining a second error rate associated with a second cluster of the plurality of clus

Assignees

• Amazon Tech Inc

Inventors

• Gupta Rahul
• Dhamala Jwala
• Verma Apurv
• Ye Qingwen
• Dabhi Mayur Himmatbhai
• Veeravanallur Srinivasan Rengarajan
• Matsoukas Spyridon
• Gens Melanie C B
• Razavi Seyed Omid
• Khatri Avni
• Natarajan Premkumar

Classifications

• G10L15/01

  Assessment or evaluation of speech recognition systems · CPC title

• G10L15/063 Primary

  Training · CPC title

• G10L2015/223

  Execution procedure of a spoken command · CPC title

• G10L2015/0631

  Creating reference templates; Clustering · CPC title

• G10L15/08

  Speech classification or search · CPC title