Method and system for improving content searching in a question and answer customer support system by using a crowd-machine learning hybrid predictive model

Patent metadata

Field	Value
Publication number	US-10467541-B2
Application number	US-201615220677-A
Country	US
Kind code	B2
Filing date	Jul 27, 2016
Priority date	Jul 27, 2016
Publication date	Nov 5, 2019
Grant date	Nov 5, 2019

Abstract

A method and system improves content searching in a question and answer customer support system by using a crowd-machine learning hybrid predictive model, according to one embodiment. The question and answer customer support system determines which customer support content to provide to users by using the hybrid predictive model, according to one embodiment. The question and answer customer support system receives a search query from a user and applies the search query (or a representation of the search query) to the hybrid predictive model, according to one embodiment. The hybrid predictive model generates a likelihood that particular customer support content is relevant to a user's search query, according to one embodiment. The question and answer customer support system acquires user feedback from users and updates/trains the hybrid predictive model based on the user feedback, according to one embodiment.

First claim

What is claimed is:

1. A computing system implemented method for using a hybrid predictive model to respond to search queries for customer support content in a question and answer customer support system, the method comprising:
storing existing customer support content data in memory allocated for use by a question and answer customer support system, the existing customer support content data representing existing customer support content entries having groups of combinations of existing search queries and existing responses, the existing search queries having been submitted to the question and answer customer support system by prior users and the existing responses having been submitted to the question and answer customer support system in response to the existing search queries;
receiving search query data from a user, the search query data representing a current search query for customer support content from the question and answer customer support system;
training a hybrid predictive model through unsupervised learning using at least a combination of machine learning content and crowdsourced user feedback to identify the customer support content of the current search query, resulting in hybrid predictive model data, the training at least including applying one or more additional natural language processing algorithms to the crowdsourced user feedback to extract one or more features of the customer support content from the user feedback, the training including consideration of a length of reference material and a style of a reference material;
applying the search query data to the predictive model data to generate content selection score data representing two or more content selection scores, at least one of the two or more content selection scores representing a relevance of the existing customer support content entries to the search query data, and at least one of the two or more content selection scores representing a quality of the customer support content, the quality-related content selection score being provided by a content quality predictive model;
selecting one of the existing customer support content entries at least partially based on the one or more content selection scores;
generating user experience display data that includes at least one user experience page and that includes the one of the existing customer support entries, the user experience display data representing a user experience display that renders the one of the existing customer support content entries in the at least one user experience page;
providing the user experience display data to the user, in response to receiving the search query data from the user;
requesting user feedback data from the user, to enable the question and answer customer support system to improve performance of the hybrid predictive model, the user feedback data representing crowdsourced user feedback related to the one of the existing customer support content entries that is included in the user experience display data; and
updating, upon receipt of user feedback data from the user, the hybrid predictive model at least partially based on the user feedback data, the received feedback including data regarding a quality of writing, accuracy and content and search ranking.

2. The computing system implemented method of claim 1 , wherein the combination of machine learning content includes features of the existing customer support content data that are identified using one or more machine learning techniques.

3. The computing system implemented method of claim 1 , wherein the crowdsourced user feedback includes features of the existing customer support content data that are extracted from the crowdsourced user feedback and/or from the user feedback data.

4. The computing system implemented method of claim 1 , further comprising:
generating the hybrid predictive model data by applying one or more predictive model training operations to:
features of the existing customer support content data that are identified by applying one or more first machine learning techniques to the existing customer support content data;
features of the existing customer support content data that are identified by applying one or more second machine learning techniques to the crowdsourced user feedback and/or to the user feedback data; and
the existing customer support content data.

5. The computing system implemented method of claim 4 , wherein the one or more predictive model training operations are selected from a group of predictive model training operations, consisting of:
regression;
logistic regression;
decision trees;
artificial neural networks;
support vector machines;
linear regression;
nearest neighbor methods;
distance based methods;
naive Bayes;
linear discriminant analysis; and
k-nearest neighbor algorithm.

6. The computing system implemented method of claim 4 , wherein features of the existing customer support content data that are identified by applying one or more first machine learning techniques to the existing customer support content data, or features of the existing customer support content data that are identified by applying one or more second machine learning techniques to the crowdsourced user feedback and/or to the user feedback data, are selected from a group of features, consisting of:
length of a search query;
length of a response;
tone and/or sentiment of a search query;
tone and/or sentiment of a response;
topic of a search query;
topic of a response;
quality of writing of a search query;
quality of writing of a response;
relevance of a search query to a user's search query;
relevance of response to a search query;
relevance of a response to a user's search query;
punctuation of a search query;
punctuation of a response;
quality of grammar of a search query or response;
perceived accuracy of a response;
a search ranking of a combination of a search query and response;
a number of responses associated with a search query;
an age of a search query;
an age of response;
a length of reference material;
a style of a reference material;
an author of a search query;
an author of a response;
an author of a reference material;
attributes of an author of a search query and/or response and/or reference material;
whether a customer support content includes a hyperlink;
a text size of a search query and/or response and/or reference material;
background and/or foreground colors and/or images used to present customer support content;
font characteristics of a customer support content;
whether a details and/or context are provided with a search query;
whether a user felt like a customer support content was helpful;
a number of other users who found a customer support content helpful;
a number of customers who viewed a customer support content;
a type of reference material; and
a number of similar questions displayed with a customer support content.

7. The computing system implemented method of claim 1 , wherein the existing customer support content entries include customer support reference materials including one or more customer support articles, customer support dictionaries, and customer support self-help guides.

8. The computing system implemented method of claim 1 , wherein the combination of machine learning content includes features of the existing customer support content data that are identified using a probabilistic topic model that is at least partially based on a Latent Dirichlet allocation algorithm.

9. The computing system implemented method of claim 1 , wherein requesting the user feedback data includes displaying feedback acquisition features in the at least one user experience page, the feedba

Assignees

• Intuit Inc

Inventors

• Podgorny Igor A
• Indyk Benjamin
• Cannon Matthew
• Guidry Jonathan
• Ma Irene F

Classifications

• G06N7/01 Primary

  Probabilistic graphical models, e.g. probabilistic networks · CPC title

• G06Q30/016 Primary

  After-sales · CPC title

• G06N20/00

  Machine learning · CPC title

• G06N5/04

  Inference or reasoning models · CPC title

• G06F16/3326

  using relevance feedback from the user, e.g. relevance feedback on documents, documents sets, document terms or passages · CPC title