Crowd worker quality improvement with continuing training

Patent metadata

Field	Value
Publication number	US-2016012372-A1
Application number	US-201414328680-A
Country	US
Kind code	A1
Filing date	Jul 10, 2014
Priority date	Jul 10, 2014
Publication date	Jan 14, 2016
Grant date	—

Abstract

Workers are trained and assisted to improve quality of task results generated by them. Initially, workers seeking to perform intelligence tasks within the context of a crowdsourcing system are provided with offline training. Subsequently, workers are qualified, with different nuances of the performance of intelligence tasks being assessed independently. A task owner specifies minimum quality thresholds, including separately specified thresholds for different nuances. Qualified workers are assigned to perform intelligence tasks, thereby producing useful work. While doing so, their quality is monitored, and online training is provided to aid workers with specific nuances of the intelligence tasks where the workers' performance is suboptimal. Workers whose quality drops below minimum quality thresholds are aided in improving their quality, including being prevented from performing intelligence tasks for a predefined period of time, being mandated to perform offline training, or combinations thereof.

First claim

We claim:

1 . A computing device for improving quality of results to intelligence tasks, the results being generated by a human worker within a crowdsourcing system, the computing device comprising one or more processing units and computer-readable media comprising computer-executable instructions that, when executed by the processing units, cause the computing device to perform steps comprising:
detecting that the quality of the results generated by the human worker are below a predetermined quality threshold; preventing, for a predetermined and finite period of time, the human worker from generating further results to intelligence tasks of a same overall task as the intelligence tasks to which the human worker generated the results that were detected to be below the predetermined quality threshold, the preventing being triggered by the detecting of the quality of the results being below the predetermined quality threshold; enabling the human worker to again perform useful work by receiving intelligence tasks and providing results thereto only in response to an expiration of the predetermined and finite period of time; detecting that the results generated by the human worker exceed the predetermined quality threshold but that the results generated by the human worker directed to at least one specific nuance of the intelligence tasks are of lesser quality than the results generated by the human worker directed other specific nuances of the intelligence tasks; and providing online training, while the human worker remains enabled to perform useful work, the online training being directed to the at least one specific nuance of the intelligence tasks, the providing being triggered by the detecting of results directed to the at least one specific nuance being of lesser quality than the results directed to the other specific nuances.

2 . The computing device of claim 1 , wherein the detecting of the quality of the results being below the predetermined quality threshold comprises estimating the quality of the results based upon a comparison between results to gold intelligence tasks generated by the human worker and predetermined definitive correct results corresponding to the gold intelligence tasks.

3 . The computing device of claim 1 , comprising further computer-executable instructions that, when executed by the processing units, cause the computing device to perform further steps comprising mandating that the human worker perform offline training during the predetermined and finite period of time; wherein the enabling is performed only in response to both the expiration of the predetermined and finite period of time and the human worker's completion of the mandated offline training.

4 . The computing device of claim 3 , wherein the mandating is based on a quantity of previous times the quality of the results generated by the human worker were detected to be below the predetermined quality threshold.

5 . The computing device of claim 1 , comprising further computer-executable instructions that, when executed by the processing units, cause the computing device to perform further steps comprising disqualifying the human worker from again performing, without subsequent qualification, useful work by receiving the intelligence tasks and providing the results thereto, the disqualifying being conditioned upon a quantity of previous times the quality of the results generated by the human worker were detected to be below the predetermined quality threshold exceeding a predetermined quantity.

6 . The computing device of claim 1 , comprising further computer-executable instructions that, when executed by the processing units, cause the computing device to perform further steps comprising:
providing training to the human worker prior to providing qualification intelligence tasks to the human worker; and qualifying the human worker to perform the useful work only if results generated by the human worker in response to the qualification intelligence tasks are of greater quality than predetermined qualification quality thresholds.

7 . The computing device of claim 6 , wherein the human worker is provided compensation for completing the training and the qualification.

8 . The computing device of claim 7 , wherein the compensation comprises increased payments for the performance of the useful work.

9 . A computing device for improving quality of results to intelligence tasks, the results being generated by a human worker within a crowdsourcing system, the computing device comprising one or more processing units and computer-readable media comprising computer-executable instructions that, when executed by the processing units, cause the computing device to perform steps comprising:
providing training to the human worker prior to providing qualification intelligence tasks to the human worker; qualifying the human worker to perform useful work, comprising generating the results to the intelligence tasks, only if results generated by the human worker in response to the qualification intelligence tasks are of greater quality than predetermined qualification quality thresholds; compensating the human worker for the performance of the useful work; and compensating, in addition to the compensation for the performance of the useful work, the human worker for completing the training and the qualification.

10 . The computing device of claim 9 , wherein the compensating the human worker for completing the training and the qualification comprises increasing the compensating the human worker for the performance of a discrete portion of the useful work.

11 . The computing device of claim 9 , wherein the compensating the human worker for completing the training and the qualification comprises progressively increasing the compensating the human worker for the performance of discrete portions of the useful work, each discrete portion being associated with progressively increased compensation.

12 . The computing device of claim 9 , comprising further computer-executable instructions that, when executed by the processing units, cause the computing device to perform further steps comprising:
detecting that the quality of the results generated by the human worker are below a predetermined quality threshold; preventing, for a predetermined and finite period of time, the human worker from generating further results to intelligence tasks of a same overall task as the intelligence tasks to which the human worker generated the results that were detected to be below the predetermined quality threshold, the preventing being triggered by the detecting of the quality of the results being below the predetermined quality threshold; enabling the human worker to again perform useful work only in response to an expiration of the predetermined and finite period of time; detecting that the results generated by the human worker exceed the predetermined quality threshold but that the results generated by the human worker directed to at least one specific nuance of the intelligence tasks are of lesser quality than the results generated by the human worker directed other specific nuances of the intelligence tasks; and providing online training, while the human worker remains enabled to perform useful work, the online training being directed to the at least one specific nuance of the intelligence tasks, the providing being triggered by the detecting of results directed to the at least one specific nuance being of lesser quality than the results directed to the other specific nuances.

13 . A method for improving quality of results to intelligence tasks, the results being generated by a human

Assignees

• Microsoft Corp

Inventors

• Patel Rajesh
• Shelford Steven
• Wang Yunling

Classifications

• G06Q10/06395 Primary

  Quality analysis or management · CPC title