Mechanical Turk integrated IDE, systems and methods

Patent metadata

Field	Value
Publication number	US-10147038-B2
Application number	US-201615230131-A
Country	US
Kind code	B2
Filing date	Aug 5, 2016
Priority date	Apr 19, 2012
Publication date	Dec 4, 2018
Grant date	Dec 4, 2018

Abstract

A Mechanical Turk-Integrated Development Environment system is disclosed. An integrated development environment (IDE) can include one or more interfaces capable of communicating with a mechanical turk engine. As a developer creates applications within the IDE, the developer can use the IDE to submit one or more requests to the mechanical turk engine. The engine constructs a mechanical turk project based on the requests and provides project tasks to workers. The results of the tasks can then be compiled and integrated back into the developer's application via the IDE. An example use includes constructing large domain specific data sets that can be applied to spoken dialog interfaces.

First claim

What is claimed is:

1. A computer implemented method of integrating a trained model trained on a training data set into an application, the method comprising:
instantiating, by a project engine, a mechanical turk project that includes a plurality of tasks available to project members;
receiving, by the project engine, project results data from the project members;
generating, by the project engine, a training data set from the project results data;
training, by the project engine, a trained model from the training data set;
converting, by an application generator module, the project results data in the form of the trained model into a trained application module formatted for integration into an application; and
causing, by the application generator module, integration of the trained application module into the application.

2. The method of claim 1 , wherein the project engine comprises at least one of the following:
an integrated development environment, a mechanical turk engine, and an application developer engine.

3. The method of claim 1 , wherein the plurality of tasks includes tasks based on at least one of the following modalities of data:
human speech, text, visual data, kinesthetic data, auditory data, and location data.

4. The method of claim 1 , wherein the plurality of tasks comprise game tasks.

5. The method of claim 4 , wherein the plurality of tasks comprise character action tasks.

6. The method of claim 1 , wherein the plurality of tasks comprise question-answer tasks.

7. The method of claim 1 , wherein the plurality of tasks comprise picture tasks.

8. The method of claim 7 , wherein the picture tasks include taking picture tasks.

9. The method of claim 1 , wherein the plurality of tasks comprise categorization tasks.

10. The method of claim 9 , wherein the categorization tasks comprise food categorization tasks.

11. The method of claim 1 , wherein the plurality of tasks comprise conceptual mapping tasks.

12. The method of claim 1 , further comprising validating the project results data.

13. The method of claim 12 , further comprising formatting the validated project results data into the training data set.

14. The method of claim 12 , wherein the mechanical turk project comprises a mechanical turk project results validation project.

15. The method of claim 14 , wherein the plurality of tasks comprise validation tasks.

16. The method of claim 1 , wherein the training data set comprises a domain-specific training set.

17. The method of claim 1 , wherein the project members comprise trained members.

18. The method of claim 17 , wherein the trained members comprises subject matter experts.

19. The method of claim 1 , wherein the project results data includes at least one of the following:
raw data, text data, audio data, image data, serial stream data, annotation data, scripts, data files, and data records.

20. The method of claim 1 , further comprising updating the project results data as time passes.

21. The method of claim 20 , wherein the step of updating the project results data includes updating the results based on usage of the application.

22. The method of claim 21 , further comprising updating the trained model based on the updated project results data.

23. The method of claim 22 , further comprising updating the application based on the updated trained model.

24. The method of claim 1 , wherein the application module comprises at least one of the following:
a source control file, a script file, an image file, an audio file, a text file, a database record, a library of callable functions, compiled code, a serialized data structure, and a domain module.

25. The method of claim 1 , wherein the step of converting the project results data includes compiling the project results data in the form of the trained model into the trained application module.

Assignees

• Nant Holdings Ip Llc

Inventors

• Ehsani Farzad
• Witt-Ehsani Silke Maren
• Master Demitrios Leo

Classifications

• G06F17/30126

  Physics · mapped topic

• G06N5/022 Primary

  Knowledge engineering; Knowledge acquisition · CPC title

• G06F17/3043

  Physics · mapped topic

• G06F8/41

  Compilation · CPC title

• G06F17/30545

  Physics · mapped topic