Result aggregation

Patent metadata

Field	Value
Publication number	US-9954942-B2
Application number	US-201314648982-A
Country	US
Kind code	B2
Filing date	Dec 11, 2013
Priority date	Dec 11, 2013
Publication date	Apr 24, 2018
Grant date	Apr 24, 2018

Abstract

Methods and systems for result aggregation in crowdsourcing are described. In one implementation, a task with a plurality of solution options is provided for workers to respond. Responses for the task are received from multiple workers, where each of the responses includes a worker response confidence value corresponding to one of the plurality of solution options. The worker response confidence value is indicative of confidence of a worker for the one of the plurality of solution options to be correct. A task answer for the task is estimated based on at least the worker response confidence value received from each of the multiple workers. A performance bonus for each of the multiple workers is computed based on the task answer, the worker response confidence value in the response from each worker, and a Proper Scoring Rule.

First claim

We claim:

1. A method for result aggregation in a crowdsourcing environment, the method comprising:
providing a task with a plurality of solution options and a range of confidence values corresponding to the plurality of solution options to multiple workers;
receiving responses for the task from the multiple workers, wherein each of the responses comprises at least a worker's selection of one of the plurality of solution options and a worker response confidence value selected from the range of confidence values corresponding to the selected one of the plurality of solution options, and wherein the worker response confidence value is indicative of confidence of a worker for the one of the plurality of solution options to be correct;
estimating a task answer for the task based on at least the worker response confidence value received from each of the multiple workers; and
computing a performance bonus for each of the multiple workers based on the task answer, the worker response confidence value in the response from each worker, and a Proper Scoring Rule.

2. The method as claimed in claim 1 , wherein the estimating the task answer is based on,
a worker capability factor for each of the multiple workers, wherein the worker capability factor is indicative of a probability of the worker to respond to a random task correctly, and
a task difficulty factor for the task, wherein the task difficulty factor is indicative of a probability of a random worker responding to the task correctly.

3. The method as claimed in claim 2 , wherein the estimating the task answer is based on boundary conditions indicative of biased nature of the multiple workers.

4. The method as claimed in claim 2 , wherein the estimating the task answer is based on boundary conditions indicative of unbiased nature of the multiple workers.

5. The method as claimed in claim 1 , wherein the estimating the task answer is based on,
a worker capability factor for each of the multiple workers, wherein the worker capability factor is indicative of a probability of the worker to respond to a random task correctly, and
a task suitability factor for the task, wherein the task suitability factor is indicative of a probability of suitability of the task for a worker.

6. The method as claimed in claim 1 , wherein at least one of the responses further comprises a plurality of worker response confidence values, each of the plurality of worker response confidence values respectively corresponding to a solution option from amongst the plurality of solution options.

7. The method as claimed in claim 1 , wherein the Proper Scoring Rule is one of a logarithmic-based Proper Scoring Rule, a quadratic-based Proper Scoring Rule, and a power-based Proper Scoring Rule.

8. A result aggregation system comprising:
a processor; and
a memory storing instructions that when executed by the processor cause the processor to:
provide a task with a plurality of solution options to multiple workers;
receive responses for the task from the multiple workers, wherein each of the responses comprises at least one worker response confidence value corresponding to at least one of the plurality of solution options, wherein the worker response confidence value is indicative of confidence of a worker for the at least one of the plurality of solution options to be correct and the worker response confidence value is selected from a range of confidence values;
estimate a task answer for the task based on at least the worker response confidence value received from each of the multiple workers; and
compute a performance bonus for each of the multiple workers based on the task answer, the worker response confidence value in the response from each worker, and a Proper Scoring Rule.

9. The result aggregation system as claimed in claim 8 , wherein the processor is to estimate the task answer based on,
a worker capability factor for each of the multiple workers, wherein the worker capability factor is indicative of a probability of the worker to respond to a random task correctly, and
a task difficulty factor for the task, wherein the task difficulty factor is indicative of a probability of a random worker responding to the task correctly.

10. The result aggregation system as claimed in claim 8 , wherein the processor is to estimate the task answer based on,
a worker capability factor for each of the multiple workers, wherein the worker capability factor is indicative of a probability of the worker to respond to a random task correctly, and
a task suitability factor for the task, wherein the task suitability factor is indicative of a probability of suitability of the task for a worker.

11. The result aggregation system as claimed in claim 10 , wherein the task answer is estimated based on boundary conditions indicative of biased nature of the multiple workers.

12. The result aggregation system as claimed in claim 8 , wherein the Proper Scoring Rule is one of a logarithmic-based Proper Scoring Rule, a quadratic-based Proper Scoring Rule, and a power-based Proper Scoring Rule.

13. A non-transitory computer-readable medium comprising instructions for result aggregation, wherein the instructions are executable by a processor to cause the processor to:
provide a task with a plurality of solution options to multiple workers via a Graphical User Interface (GUI) that includes an input mechanism that receives worker response confidence values corresponding to the plurality of solution options;
receive responses for the task from the multiple workers, wherein each of the responses comprises an input for the worker response confidence value corresponding to a selected one of the plurality of solution options, and wherein the worker response confidence value is indicative of confidence of a worker for the one of the plurality of solution options to be correct;
estimate a task answer for the task based on at least the worker response confidence value received from each of the multiple workers; and
compute a performance bonus for each of the multiple workers based on the task answer, the worker response confidence value in the response from each worker, and a Proper Scoring Rule.

14. The non-transitory computer-readable medium as claimed in claim 13 , wherein the instructions are to cause the processor to estimate the task answer based on,
a worker capability factor for each of the multiple workers, wherein the worker capability factor is indicative of a probability of the worker to respond to a random task correctly, and
a task difficulty factor for the task, wherein the task difficulty factor is indicative of a probability of a random worker responding to the task correctly.

15. The non-transitory computer-readable medium as claimed in claim 13 , wherein the instructions are to cause the processor to estimate the task answer based on,
a worker capability factor for each of the multiple workers, wherein the worker capability factor is indicative of a probability of the worker to respond to a random task correctly, and
a task suitability factor for the task, wherein the task suitability factor is indicative of a probability of suitability of the task for a worker.

16. The non-transitory computer-readable medium as claimed in claim 13 , wherein the input mechanism provided on the GUI comprises slider type inputs.

17. The non-transitory computer-readable medium as claimed in claim 13 , wherein the instructions are to cause the processor to:
for each of the responses:
set a system response confidence value for the selected option as equal to the worker respon

Assignees

• Hewlett Packard Development Co
• Entit Software Llc

Inventors

• Dey Prasenjit
• Chandra Praphul
• Gupta Amit
• Arun Kalyanasundaram

Classifications

• G06Q10/40

  Business processes related to social networking or social networking services · CPC title

• G06N7/01

  Probabilistic graphical models, e.g. probabilistic networks · CPC title

• H04L67/104 Primary

  Peer-to-peer [P2P] networks · CPC title

• G06N7/005

  Physics · mapped topic

• G06F17/30654

  Physics · mapped topic