System and method for image annotation

Patent metadata

Field	Value
Publication number	US-10360257-B2
Application number	US-201715672217-A
Country	US
Kind code	B2
Filing date	Aug 8, 2017
Priority date	Aug 8, 2017
Publication date	Jul 23, 2019
Grant date	Jul 23, 2019

Abstract

A system and method for implementing an image annotation platform are disclosed. A particular embodiment includes: registering a plurality of labelers to which annotation tasks are assigned; assigning annotation tasks to the plurality of labelers; determining if the annotation tasks can be closed or re-assigned to the plurality of labelers; aggregating annotations provided by the plurality of labelers as a result of the closed annotation tasks; evaluating a level of performance of the plurality of labelers in providing the annotations; and calculating payments for the plurality of labelers based on the quantity and quality of the annotations provided by the plurality of labelers.

First claim

What is claimed is:

1. A system comprising:
a data processor; and
an image annotation module, executable by the data processor, the image annotation module being configured to perform an image annotation operation, the image annotation operation being configured to:
register a plurality of labelers to which annotation tasks are assigned;
assign annotation tasks to the plurality of labelers;
randomly assigning evaluation tasks to the plurality of labelers, the evaluation tasks configured to evaluate the quality of the annotations provided by the plurality of labelers;
enabling the plurality of labelers in an annotation verification chain to add, delete, or modify annotations provided by sequentially prior labelers;
determine if the annotation tasks can be closed or re-assigned to the plurality of labelers;
aggregate annotations provided by the plurality of labelers as a result of the closed annotation tasks;
evaluate a level of performance of the plurality of labelers in providing the annotations; and
calculate payments for the plurality of labelers based on the quantity and quality of the annotations provided by the plurality of labelers.

2. The system of claim 1 wherein the plurality of labelers are in networked data communication with the data processor.

3. The system of claim 1 wherein at least one of the assigned annotation tasks is an evaluation task.

4. The system of claim 1 being further configured to enable each of the plurality of labelers to add, delete, or modify annotations provided by prior labelers.

5. The system of claim 1 being further configured to determine if an annotation task needs to be re-assigned to another labeler.

6. The system of claim 1 wherein the annotations correspond to object labels identified in an input image.

7. The system of claim 1 being further configured to receive images from a video feed from an autonomous vehicle.

8. A method comprising:
registering a plurality of labelers to which annotation tasks are assigned;
assigning annotation tasks to the plurality of labelers;
randomly assigning evaluation tasks to the plurality of labelers, the evaluation tasks configured to evaluate the quality of the annotations provided by the plurality of labelers;
enabling the plurality of labelers in an annotation verification chain to add, delete, or modify annotations provided by sequentially prior labelers;
determining if the annotation tasks can be closed or re-assigned to the plurality of labelers;
aggregating annotations provided by the plurality of labelers as a result of the closed annotation tasks;
evaluating a level of performance of the plurality of labelers in providing the annotations; and
calculating payments for the plurality of labelers based on the quantity and quality of the annotations provided by the plurality of labelers.

9. The method of claim 8 wherein the plurality of labelers are in networked data communication.

10. The method of claim 8 wherein at least one of the assigned annotation tasks is an evaluation task.

11. The method of claim 8 including enabling each of the plurality of labelers to add, delete, or modify annotations provided by prior labelers.

12. The method of claim 8 including determining if an annotation task needs to be re-assigned to another labeler.

13. The method of claim 8 wherein the annotations correspond to object labels identified in an input image.

14. The method of claim 8 including receiving images from a video feed from an autonomous vehicle.

15. A non-transitory machine-useable storage medium embodying instructions which, when executed by a machine, cause the machine to:
register a plurality of labelers to which annotation tasks are assigned;
assign annotation tasks to the plurality of labelers;
randomly assign evaluation tasks to the plurality of labelers, the evaluation tasks configured to evaluate the quality of the annotations provided by the plurality of labelers;
enable the plurality of labelers in an annotation verification chain to add, delete, or modify annotations provided by sequentially prior labelers;
determine if the annotation tasks can be closed or re-assigned to the plurality of labelers;
aggregate annotations provided by the plurality of labelers as a result of the closed annotation tasks;
evaluate a level of performance of the plurality of labelers in providing the annotations; and
calculate payments for the plurality of labelers based on the quantity and quality of the annotations provided by the plurality of labelers.

16. The machine-useable storage medium of claim 15 wherein the plurality of labelers are in networked data communication with the data processor.

17. The machine-useable storage medium of claim 15 wherein at least one of the assigned annotation tasks is an evaluation task.

18. The machine-useable storage medium of claim 15 wherein the instructions being further configured to enable each of the plurality of labelers to add, delete, or modify annotations provided by prior labelers.

19. The machine-useable storage medium of claim 15 wherein the instructions being further configured to determine if an annotation task needs to be re-assigned to another labeler.

20. The machine-useable storage medium of claim 15 wherein the instructions being further configured to receive images from a video feed from an autonomous vehicle.

Assignees

• TuSimple

Inventors

• Hou Xiaodi
• Liu Siyuan
• Zhou Kai

Classifications

• G06F40/169 Primary

  Annotation, e.g. comment data or footnotes · CPC title

• G06Q30/0283

  Price estimation or determination · CPC title

• G06F16/583

  using metadata automatically derived from the content · CPC title

• G06Q10/063112

  Skill-based matching of a person or a group to a task · CPC title

• G06F16/58 Primary

  Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually · CPC title