Clarification of natural language requests using neural networks

Patent metadata

Field	Value
Publication number	US-11610064-B2
Application number	US-201916582700-A
Country	US
Kind code	B2
Filing date	Sep 25, 2019
Priority date	Dec 10, 2018
Publication date	Mar 21, 2023
Grant date	Mar 21, 2023

Abstract

A user of an automated natural language system may submit an ambiguous or incomplete request, and interactive techniques may be used to obtain clarification information from the user and then determine a response for presentation to the user. A user's initial request may be processed to compute a category score for each possible category of request. The category scores may be processed to determine if clarification of the request is needed. Where clarification is needed, one or more tags may be selected to determine a clarification question to be presented to the user. For example, a tag clarification score may be computed for each tag that indicates a value of the tag in clarifying the request. After receiving the clarification information from the user, one or more categories may be selected or, where needed, additional clarification information may be obtained.

First claim

What is claimed is:

1. A computer-implemented method for clarifying a request received from a user, comprising:
receiving the request from the user, wherein the request comprises natural language;
computing a sequence of embedding vectors from text of the request, wherein the embedding vectors represent the text in a vector space;
processing the sequence of embedding vectors to compute category scores wherein:
the category scores are determined by processing the sequence of embedding vectors with one or more neural network layers and a classifier,
each category score corresponds to a category of a plurality of categories and indicates a match between the request and the corresponding category, and
each category of the plurality of categories corresponds to one or more tags of a plurality of tags;

determining, using the category scores, to obtain clarification of the request from the user;
computing tag clarification scores using the category scores wherein each tag clarification score corresponds to a tag of the plurality of tags and indicates a value of the corresponding tag in clarifying the request;
determining a clarifying question using the tag clarification scores;
presenting the clarifying question to the user;
receiving clarification information from the user in response to the clarifying question;
computing second category scores using the clarification information; and
selecting a category for the request using the second category scores.

2. The computer-implemented method of claim 1 , wherein the plurality of categories correspond to (i) intents of an automated system for processing natural language or (ii) information items of an information retrieval system.

3. The computer-implemented method of claim 1 , wherein computing a tag clarification score for a tag comprises computing:
a probability of the tag being true given the text of the request;
an entropy of the category scores given the text of the request and the tag being true;
a probability of the tag being false given the text of the request; and
an entropy of the category scores given the text of the request and the tag being false.

4. The computer-implemented method of claim 1 , wherein determining the clarifying question comprises:
selecting a tag by comparing the tag clarification scores to a threshold; and
determining the clarifying question using the selected tag.

5. The computer-implemented method of claim 4 , wherein determining the clarifying question comprises determining that the selected tag is multiple choice.

6. The computer-implemented method of claim 1 , wherein computing the second category scores comprises processing text corresponding to the clarification information with the neural network.

7. The computer-implemented method of claim 1 , wherein the clarification information from the user comprises natural language input and computing the second category scores comprises processing text of the natural language input.

8. The computer-implemented method of claim 1 , comprising:
computing tag truth scores wherein each tag truth score corresponds to a tag and indicates a match between the tag and the request; and
determining the clarifying question using the tag truth scores.

9. The computer-implemented method of claim 1 , wherein the classifier comprises one or more of recurrent neural network layers.

10. A system for clarifying a request, the system comprising:
at least one server computer comprising at least one processor and at least one memory, the at least one server computer configured to:
receive the request from a user, wherein the request comprises natural language;
compute a sequence of embedding vectors from text of the request, wherein the embedding vectors represent the text in a vector space;
process the sequence of embedding vectors, to compute category scores wherein:
the category scores are determined by processing the sequence of embedding vectors with one or more neural network layers and a classifier,
each category score corresponds to a category of a plurality of categories and indicates a match between the request and the corresponding category, and
each category of the plurality of categories corresponds to one or more tags of a plurality of tags;

determine, using the category scores, to obtain clarification of the request from the user;
compute tag clarification scores using the category scores wherein each tag clarification score corresponds to a tag of the plurality of tags and indicates a value of the corresponding tag in clarifying the request;
determine a clarifying question using the tag clarification scores;
present the clarifying question to the user;
receive clarification information from the user in response to the clarifying question;
compute second category scores using the clarification information; and
select a category using the second category scores.

11. The system of claim 10 , wherein the at least one server computer is configured to determine the clarifying question by selecting a question template from a plurality of question templates.

12. The system of claim 10 , wherein the user is a customer of a company seeking customer support from the company.

13. The system of claim 10 , wherein the neural network is a recurrent neural network.

14. The system of claim 10 , wherein a tag comprises a word or a phrase.

15. The system of claim 10 , wherein the at least one server computer is configured to determine to request clarification of the request by comparing a maximum category score to a threshold.

16. The system of claim 10 , wherein the at least one server computer is configured to compute a tag clarification score for a tag by computing an expected change in entropy of the category scores.

17. One or more non-transitory, computer-readable media comprising computer-executable instructions that, when executed, cause at least one processor to perform actions comprising:
receiving a request from a user, wherein the request comprises natural language;
computing a sequence of embedding vectors from text of the request, wherein the embedding vectors represent the text in a vector space;
processing the sequence of embedding vectors to compute category scores wherein:
the category scores are determined by processing the sequence of embedding vectors with one or more neural network layers and a classifier,
each category score corresponds to a category of a plurality of categories and indicates a match between the request and the corresponding category, and
each category of the plurality of categories corresponds to one or more tags of a plurality of tags;

determining, using the category scores, to obtain clarification of the request from the user;
computing tag clarification scores using the category scores wherein each tag clarification score corresponds to a tag of the plurality of tags and indicates a value of the corresponding tag in clarifying the request;
determining a clarifying question using the tag clarification scores;
computing a request vector by processing text of the request with the neural network;
obtaining for each category of the plurality of categories, a category vector computed by the neural network; and
performing a computation using the request vector and the category vectors.

18. The one or more non-transitory, computer-readable media of claim 17 , wherein:
the plurality of categories correspond to information items of an information retrieval system; and
a category vector for a category is computed by processing text of a corresponding in

Assignees

• Asapp Inc

Inventors

• Yu Lili
• Lei Tao
• Chen Howard
• Wang Sida

Classifications

• G06N3/092

  Reinforcement learning · CPC title

• G06N3/0442

  characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06Q30/016 Primary

  After-sales · CPC title

• G06F16/285

  Clustering or classification · CPC title