Providing semantically relevant answers to questions

Patent metadata

Field	Value
Publication number	US-10789944-B2
Application number	US-201816147473-A
Country	US
Kind code	B2
Filing date	Sep 28, 2018
Priority date	Jun 3, 2018
Publication date	Sep 29, 2020
Grant date	Sep 29, 2020

Abstract

A device implementing a system for determining whether a semantically relevant answer can be provided with respect to a new question includes a processor configured to identify a question and a semantically relevant answer from source data, and to identify a semantically irrelevant answer to the question from a corpus of data corresponding to the source data based at least in part on the question and the answer. The processor is configured to assign a positive label to the semantically relevant answer, and a negative label to the semantically irrelevant answer. The processor is configured to generate a machine learning model based on the question, the positive label assigned to the semantically relevant answer, and the negative label assigned to the semantically irrelevant answer, and to provide the machine learning model to facilitate a determination of whether a semantically relevant answer can be provided with respect to a subsequent question.

First claim

What is claimed is:

1. A device, comprising:
at least one processor; and
a memory including instructions that, when executed by the at least one processor, cause the at least one processor to:
identify a question and an answer to the question based on source data, the answer being determined to be semantically relevant with respect to the question based at least in part on the answer including terms that appear in a particular order with respect to an order that terms appear in the question;
perform a search on a corpus of data corresponding to the source data, based at least in part on the question and the answer to obtain another answer to the question, the other answer being determined to be semantically irrelevant with respect to the question;
assign a positive label to the answer that is semantically relevant with respect to the question, and a negative label to the other answer that is semantically irrelevant with respect to the question;
generate a machine learning model based at least in part on the question, the positive label assigned to the answer that is semantically relevant, and the negative label assigned to the other answer that is semantically irrelevant; and
provide the machine learning model to facilitate a determination of whether a likely semantically relevant answer can be provided with respect to a subsequent question.

2. The device of claim 1 , wherein the instructions further cause the at least one processor to:
receive the subsequent question;
search for candidate answers to the subsequent question; and
provide the subsequent question and the candidate answers to the machine learning model to obtain a confidence score for each of the candidate answers.

3. The device of claim 2 , wherein the instructions further cause the at least one processor to:
determine that the likely semantically relevant answer can be provided with respect to the subsequent question when the confidence score for at least one of the candidate answers satisfies a confidence score threshold;
select, in response to the determination, a candidate answer from the at least one of the candidate answers; and
provide the selected candidate answer in response to the subsequent question.

4. The device of claim 2 , wherein the instructions further cause the at least one processor to:
determine that the likely semantically relevant answer cannot be provided with respect to the subsequent question when the confidence score for each of the candidate answers does not satisfy a confidence score threshold; and
provide, in response to the determination, an indication that an answer cannot be provided with respect to the subsequent question.

5. The device of claim 2 , wherein searching for the candidate answers comprises performing an internet-based search with the subsequent question.

6. The device of claim 2 , wherein the subsequent question is received via a virtual assistant application, a browser, or a system-wide search application.

7. The device of claim 1 , wherein the source data comprises question and answer forums, and wherein the corpus of data is generated based on crawling the question and answer forums.

8. The device of claim 1 , wherein the other answer is determined to be semantically related to the question and the answer.

9. A computer program product comprising code stored in a non-transitory computer-readable storage medium, the code comprising:
code to identify a question and an answer to the question based on source data, the answer being determined to be semantically relevant with respect to the question based at least in part on the answer including terms that appear in a particular order with respect to an order that terms appear in the question;
code to perform a search on a corpus of data corresponding to the source data, based at least in part on the question and the answer to obtain other answers to the question, the other answers being determined to be semantically irrelevant with respect to the question;
code to assign a positive label to the answer that is determined to be semantically relevant with respect to the question, and negative labels to the other answers that are determined to be semantically irrelevant with respect to the question;
code to generate a machine learning model based at least in part on the question, the positive label assigned to the answer that is determined to be semantically relevant, and the negative labels assigned to the other answers that are determined to be semantically irrelevant; and
code to provide the machine learning model to facilitate a determination of whether a likely semantically relevant answer can be provided with respect to a subsequent question.

10. The computer program product of claim 9 , wherein a total number of the other answers is predefined.

11. The computer program product of claim 9 , the code further comprising:
code to receive the subsequent question;
code to search for candidate answers to the subsequent question; and
code to provide the subsequent question and the candidate answers to the machine learning model to obtain a confidence score for each of the candidate answers.

12. The computer program product of claim 11 , the code further comprising:
code to determine that the likely semantically relevant answer can be provided with respect to the subsequent question when the confidence score for at least one of the candidate answers satisfies a confidence score threshold;
code to select, in response to the determination, a candidate answer from the at least one of the candidate answers; and
code to provide the selected candidate answer in response to the subsequent question.

13. The computer program product of claim 11 , the code further comprising:
code to determine that the likely semantically relevant answer cannot be provided with respect to the subsequent question when the confidence score for each of the candidate answers does not satisfy a confidence score threshold; and
code to provide, in response to the determination, an indication that an answer cannot be provided with respect to the subsequent question.

14. The computer program product of claim 11 , wherein searching for the candidate answers comprises performing an internet-based search with the subsequent question.

15. The computer program product of claim 11 , wherein the subsequent question is received via a virtual assistant application, a browser, or a system-wide search application.

16. The computer program product of claim 9 , wherein the source data comprises question and answer forums, and wherein the corpus of data is generated based on crawling the question and answer forums.

17. The computer program product of claim 9 , wherein at least one of the other answers is semantically related to the question and the answer.

18. A method, comprising:
identifying a question and an answer to the question based on source data, the answer being determined to be semantically relevant with respect to the question based at least in part on the answer including terms that appear in a particular order with respect to an order that terms appear in the question;
perform a search on a corpus of data corresponding to the source data, based at least in part on the question and the answer to obtain another answer to the question, the other answer being determined to be semantically irrelevant with respect to the question;
assigning a positive label to the answer that is determined to be semantically relevant with respect to the question, and a negative label to the other answer that is determined t

Assignees

• Apple Inc

Inventors

• Siva Kumaran Saravana Kumar
• Yan Su
• Gholamzadeh Ashkan

Classifications

• G06N20/00

  Machine learning · CPC title

• G10L15/1815 Primary

  Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning · CPC title

• G10L15/22

  Procedures used during a speech recognition process, e.g. man-machine dialogue · CPC title

• G06F16/2423

  Interactive query statement specification based on a database schema · CPC title

• G06F16/951

  Indexing; Web crawling techniques · CPC title