Scalable ground truth disambiguation

Patent metadata

Field	Value
Publication number	US-11657104-B2
Application number	US-201916659165-A
Country	US
Kind code	B2
Filing date	Oct 21, 2019
Priority date	Apr 18, 2017
Publication date	May 23, 2023
Grant date	May 23, 2023

Abstract

Methods, computer program products, and systems are presented. The methods include, for instance: obtaining an utterance input from a user agent, and collecting context data of the utterance input. A context tag is generated based on the context data, and one or more ground truth having respective utterance semantically identical to the utterance input is selected. Semantical relationship between the context tag and an intent of the selected ground truth is examined and the selected ground truth is updated with the context tag.

First claim

What is claimed is:

1. A computer implemented method comprising:
obtaining an utterance input from a user agent;
collecting context data of the utterance input from the user agent, wherein the context data describes circumstances of the utterance input;
generating a context tag based on the context data, wherein the context tag corresponds to the utterance input;
selecting a ground truth, wherein the selecting includes using the utterance input and the context tag, wherein the ground truth includes an utterance and an intent, wherein the utterance of the ground truth is semantically identical to the utterance input, and wherein the intent of the ground truth is semantically consistent with the context tag; and
updating the ground truth, wherein the updating includes attaching the context tag, wherein the updating the ground truth includes updating a first ground truth so that the first ground truth includes the context tag, and training a machine learning process using first training data, wherein the first training data used to train the machine learning process includes the first ground truth tagged with the context tag and having a first utterance and a first intent.

2. The computer implemented method of claim 1 , wherein the selecting includes invoking the machine learning process with the utterance input and the context tag so that the machine learning process provides a first ground truth having a first utterance and a first intent.

3. The computer implemented method of claim 1 , wherein the context data is absent of data derived from the utterance input, and wherein the context data of the utterance includes manually input data, metadata of a page from which the utterance input has been provided, and login information of a user.

4. The computer implemented method of claim 1 , wherein the context data is absent of data derived from the utterance input, and wherein the context data of the utterance includes manually input data, and metadata of a page from which the utterance input has been provided.

5. The computer implemented method of claim 1 , wherein the context data is absent of data derived from the utterance input, and wherein the context data of the utterance includes metadata of a page from which the utterance input has been provided, and login information of a user.

6. The computer implemented method of claim 1 , further including updating training data for training the machine learning process so that the training data includes the ground truth as tagged with the context tag.

7. The computer implemented method of claim 1 , wherein the user agent runs in a retail website, and wherein the context data of the utterance includes a department name of a page from which the utterance input has been provided, a product type of the page, previous search terms used by a user, and search details selected by the user.

8. The computer implemented method of claim 1 , wherein the context data is absent of data derived from the utterance input, and wherein the context data of the utterance includes manually input data, metadata of a page from which the utterance input has been provided, and login information of a user, wherein the user agent runs in a retail website, and wherein the context data of the utterance includes a department name of a page from which the utterance input has been provided, a product type of the page, previous search terms used by a user, and search details selected by the user, wherein the generating comprises selecting an instance from the context data for the context tag, wherein the instance is associated with a first intent of the utterance input, wherein the instance distinguishes the first intent from a second intent, wherein the utterance input means both the first intent and the second intent, and assigning the context tag for the utterance input with the instance, wherein the selecting comprises discovering one or more ground truth that has respective utterance identical to the utterance input, ascertaining that an intent of a certain ground truth from the discovering is semantically relevant to the context tag by examining respective intent of the one or more ground truth from the discovering, and determining the certain ground truth as a ground truth corresponding to the utterance input and the context tag.

9. The computer implemented method of claim 1 , wherein the context data of the utterance is selected from the group consisting of: a manual input, metadata of a page from which the utterance input has been provided, and login information of a user.

10. The computer implemented method of claim 1 , wherein the user agent runs in a retail website, and wherein the context data of the utterance input is selected from the group consisting of: a department name of a page from which the utterance input has been provided, a product type of the page, previous search terms used by a user, and search details selected by the user.

11. The computer implemented method of claim 1 , the selecting comprising: discovering one or more ground truth that has respective utterance identical to the utterance input; ascertaining that an intent of a first ground truth from the discovering is semantically relevant to the context tag by examining respective intent of the one or more ground truth from the discovering; and determining the first ground truth as a ground truth corresponding to the utterance input and the context tag.

12. The computer implemented method of claim 1 , wherein the method includes training the machine learning process in dependence on the updating.

13. The computer implemented method of claim 1 , wherein the method includes training the machine learning process using the updated ground truth.

14. The computer implemented method of claim 1 , wherein the method includes invoking the machine learning process with the utterance input.

15. A computer program product comprising:
a computer readable storage medium readable by one or more processing circuit and storing instructions for execution by one or more processor for performing a method comprising:
obtaining an utterance input from a user agent;
collecting context data of the utterance input from the user agent, wherein the context data describes circumstances of the utterance input;
generating a context tag based on the context data, wherein the context tag corresponds to the utterance input;
selecting a ground truth, wherein the selecting includes using the utterance input and the context tag, wherein the ground truth includes an utterance and an intent, wherein the utterance of the ground truth is semantically identical to the utterance input, and wherein the intent of the ground truth is semantically consistent with the context tag; and
updating the ground truth, wherein the updating includes attaching the context tag, wherein the updating the ground truth includes updating a first ground truth so that the first ground truth includes the context tag, and training a machine learning process using first training data, wherein the first training data used to train the machine learning process includes the first ground truth tagged with the context tag and having a first utterance and a first intent.

16. The computer program product of claim 15 , wherein the collecting includes processing web browsing data of a user to extract the context data.

17. A system comprising:
a memory;
one or more processor in communication with the memory; and
program instructions executable by the one or more processor via the memory to perform a method comprising:
obtaining an utterance input from a user agent;

Assignees

• IBM

Inventors

• Altaf Faheem
• Deluca Lisa Seacat
• Srinivas Raghuram

Classifications

• G06F16/953 Primary

  Querying, e.g. by the use of web search engines · CPC title

• G06F40/20

  Natural language analysis (semantic analysis of natural language G06F40/30) · CPC title

• G06N5/025

  Extracting rules from data · CPC title

• G06N20/00

  Machine learning · CPC title