System and method for human-machine hybrid prediction of events

Patent metadata

Field	Value
Publication number	US-11625562-B2
Application number	US-201916708166-A
Country	US
Kind code	B2
Filing date	Dec 9, 2019
Priority date	Feb 11, 2019
Publication date	Apr 11, 2023
Grant date	Apr 11, 2023

Abstract

A method for generating human-machine hybrid predictions of answers to forecasting problems includes: parsing text of an individual forecasting problem to identify keywords; generating machine models based on the keywords; scraping data sources based on the keywords to collect scraped data relevant to the individual forecasting problem; providing the scraped data to the machine models; receiving machine predictions of answers to the individual forecasting problem from the machine models based on the scraped data; providing, by the computer system via a user interface, the scraped data to human participants; receiving, by the computer system via the user interface, human predictions of answers to the individual forecasting problem from the human participants; aggregating the machine predictions with the human predictions to generate aggregated predictions; and generating and outputting a hybrid prediction based on the aggregated predictions.

First claim

What is claimed is:

1. A method for generating human-machine hybrid predictions of answers to forecasting problems, comprising:
parsing, by a computer system, text of an individual forecasting problem (IFP) to identify a plurality of keywords;
generating, by the computer system, one or more machine learning models based on the keywords;
scraping, by the computer system, one or more data sources based on the keywords to collect scraped data relevant to the individual forecasting problem;
training, by the computer system, the one or more machine learning models based on the scraped data;
receiving, by the computer system, one or more machine predictions of answers to the individual forecasting problem from the one or more machine learning models based on the scraped data;
providing, by the computer system via a user interface, the scraped data to one or more human participants;
receiving, by the computer system via the user interface, one or more human predictions of answers to the individual forecasting problem from the one or more human participants;
aggregating, by the computer system, the one or more machine predictions with the one or more human predictions to generate aggregated predictions;
generating and outputting, by the computer system, a hybrid prediction based on weighting the one or more machine predictions and the one or more human predictions of the aggregated predictions;
scraping, by the computer system, the one or more data sources based on the keywords to collect new scraped data relevant to the individual forecasting problem;
detecting a significant change in the new scraped data compared to the scraped data; and
in response to detecting the significant change in the new scraped data compared to the scraped data:
alerting the one or more human participants regarding the new scraped data;
displaying the new scraped data to the one or more human participants; and
receiving one or more updated human predictions from the one or more human participants.

2. The method of claim 1 , further comprising:
providing the keywords to the human participants via the user interface; and
receiving votes on the keywords from the human participants via the user interface,
wherein the scraping the data sources is performed in accordance with the votes on the keywords.

3. The method of claim 1 , further comprising displaying a machine prediction computed by a machine learning model of the one or more machine learning models to a human participant of the one or more human participants via the user interface.

4. The method of claim 1 , further comprising:
receiving one or more human participant rationales, each corresponding to one of the one or more human predictions; and
displaying the one or more human participant rationales in the user interface in association with the corresponding human predictions.

5. The method of claim 1 , further comprising, in response to detecting the significant change in the new scraped data compared to the scraped data:
providing the new scraped data to further train the one or more machine learning models; and
receiving one or more updated machine predictions from the one or more machine learning models based on the new scraped data.

6. The method of claim 1 , wherein the individual forecasting problem is assigned to the one or more human participants of a plurality of human participants by:
computing a participant skill vector for each of the plurality of human participants based on a plurality of prior responses to a plurality of prior individual forecasting problems;
comparing the participant skill vector of each of the plurality of human participants to the individual forecasting problem to compute a plurality of participant-IFP match scores;
identifying a plurality of participant-IFP match scores satisfying a threshold; and
assigning the individual forecasting problem to the one or more human participants, wherein the participant-IFP match scores corresponding to the one or more human participants satisfy the threshold.

7. The method of claim 1 , wherein the generating the hybrid prediction comprises:
applying weights to the one or more human predictions and the one or more machine predictions to compute weighted human predictions and weighted machine predictions; and
computing the hybrid prediction based on a combination of the weighted human predictions and the weighted machine predictions.

8. The method of claim 7 , wherein the computing the hybrid prediction comprises supplying the one or more human predictions and the one or more machine predictions to an artificial neural network configured with the weights.

9. The method of claim 7 , further comprising:
receiving, by the computer system from the one or more machine learning models, one or more machine confidence scores, each of the machine confidence scores corresponding to one of the one or more machine predictions; and
receiving, by the computer system via the user interface, from the one or more human participants, one or more participant confidence scores, each of the participant confidence scores corresponding to one of the one or more human predictions,
wherein each of the weights associated with the one or more machine predictions is computed based on a corresponding one of the machine confidence scores, and
wherein each of the weights associated with the one or more human predictions is computed based on a corresponding one of the participant confidence scores.

10. A computer system for generating human-machine hybrid predictions of answers to forecasting problems, comprising:
one or more processors; and
one or more computer memory units storing instructions that, when executed by the one or more processors, cause the one or more processors to:
parse text of an individual forecasting problem (IFP) to identify a plurality of keywords;
generate one or more machine learning models based on the keywords;
scrape one or more data sources based on the keywords to collect scraped data relevant to the individual forecasting problem;
train the one or more machine learning models based on the scraped data;
receive one or more machine predictions of answers to the individual forecasting problem from the one or more machine learning models based on the scraped data;
provide, via a user interface in communication with the one or more processors, the scraped data to one or more human participants;
receive, via the user interface, one or more human predictions from the one or more human participants;
aggregate the one or more machine predictions with the one or more human predictions to generate aggregated predictions; and
generate and output a hybrid prediction based on weighting the one or more machine predictions and the one or more human predictions of the aggregated predictions,

wherein the individual forecasting problem is assigned to the one or more human participants of a plurality of human participants by:
computing a participant skill vector for each of the plurality of human participants based on a plurality of prior responses to a plurality of prior individual forecasting problems;
comparing the participant skill vector of each of the plurality of human participants to the individual forecasting problem to compute a plurality of participant-IFP match scores;
identifying a plurality of participant-IFP match scores satisfying a threshold; and
assigning the individual forecasting problem to the one or more human participants, wherein the participant-IFP match scores corresponding to the one or more human participants satisfy the threshold.

11. The computer system of claim 10 , wherein the one or more computer memory units further stor

Assignees

• Hrl Lab Llc

Inventors

• Huber David J
• Lu Tsai-Ching
• Stepp Nigel D
• Jammalamadaka Aruna
• Kim Hyun J
• Johnson Samuel D

Classifications

• G06Q10/40

  Business processes related to social networking or social networking services · CPC title

• G06N3/082

  modifying the architecture, e.g. adding, deleting or silencing nodes or connections · CPC title

• G06N3/092

  Reinforcement learning · CPC title

• G06F40/205

  Parsing · CPC title

• G06F18/251

  of input or preprocessed data · CPC title