Benchmarking machine learning models via performance feedback

Patent metadata

Field	Value
Publication number	US-10949252-B1
Application number	US-201815895747-A
Country	US
Kind code	B1
Filing date	Feb 13, 2018
Priority date	Feb 13, 2018
Publication date	Mar 16, 2021
Grant date	Mar 16, 2021

Abstract

Techniques for benchmarking a machine learning model/algorithm are described. For example, in some instances a method includes generating an execution plan for benchmarking of at least one task corresponding to a machine learning model based on an identified machine learning model, identified training data, and at least one objective for the benchmarking job; receiving execution statistics about the execution of the task as a part of the benchmarking job according to the execution plan; and updating the execution plan based at least in part on the received execution statistics of the task.

First claim

What is claimed is:

1. A computer-implemented method comprising:
receiving a request to generate an execution plan for a benchmarking job of at least one training or inference task for an identified machine learning model, the request including at least one objective for the benchmarking job;
generating the execution plan for benchmarking job the at least one training or inference task, the execution plan corresponding to a machine learning model based on the identified machine learning model, identified training data, and the at least one objective for the benchmarking job;
providing the generated execution plan to a monitor service that is to receive statistics about an execution of the execution plan;
receiving execution statistics about the execution of the at least one training or inference task performed by the machine learning model as a part of the benchmarking job according to the execution plan from the monitor service; and
updating the execution plan based at least in part on the received execution statistics about the execution of the at least one training or inference task.

2. The computer-implemented method of claim 1 , further comprising:
triggering execution of the execution plan by the monitor service.

3. The computer-implemented method of claim 1 , wherein the generating of the execution plan is further based on at least one of hardware available to a user requesting the benchmarking job, available jobs to batch with the task, previous execution of the task to be run as a job, and previous execution of similar jobs.

4. A computer-implemented method comprising:
generating an execution plan for a benchmarking job of at least one training or inference task corresponding to a machine learning model based on an identified machine learning model, identified training data, and at least one objective for the benchmarking job;
receiving execution statistics about the execution of the at least one training or inference task performed by the machine learning model as a part of the benchmarking job according to the execution plan; and
updating the execution plan based at least in part on the received execution statistics of the at least one training or inference task.

5. The computer-implemented method of claim 4 , further comprising:
triggering execution of the execution plan by a monitor service.

6. The computer-implemented method of claim 4 , wherein the generating of the execution plan is further based on at least one of hardware available to a user requesting the benchmarking job, available jobs to batch with the task, previous execution of the task to be run as a job, and previous execution of similar jobs.

7. The computer-implemented method of claim 4 , wherein the generating of the execution plan includes simulating the training of a model of the task to determine what hardware is capable of executing the task and meet the objective.

8. The computer-implemented method of claim 4 , wherein the generating of the execution plan includes executing the training of a model of the task to determine what hardware is capable of executing the task and meet the objective.

9. The computer-implemented method of claim 4 , wherein the execution of the task as a part of the benchmarking job according to the execution plan occurs in a virtual machine.

10. The computer-implemented method of claim 4 , wherein the execution of the task as a part of the benchmarking job according to the execution plan occurs in a container.

11. The computer-implemented method of claim 4 , wherein the objective is at least one of: a shortest possible time to complete the task, a minimum hardware usage possible to complete the task, and accuracy of the machine learning model.

12. The computer-implemented method of claim 4 , wherein the generated execution plan is a batch that includes other jobs to be executed on the same hardware.

13. The computer-implemented method of claim 4 , further comprising:
generating metrics about the execution of the task based on the received execution statistics.

14. The computer-implemented method of claim 4 , further comprising:
generating an alarm when the execution of the task exceeds a threshold metric.

15. A system comprising:
execution resources to implement a benchmarking job according to an execution plan; and
a benchmarking service implemented by a second one or more electronic devices, the benchmarking service including instructions that upon execution cause the benchmarking service to:
generate an execution plan for a benchmarking job of at least one training or inference task corresponding to a machine learning model based on an identified machine learning model, identified training data, and at least one objective for the benchmarking job;
receive execution statistics about the execution of the at least one training or inference task performed by the machine learning model as a part of the benchmarking job according to the execution plan; and
update the execution plan based at least in part on the received execution statistics of the at least one training or inference task.

16. The system of claim 15 , wherein the instructions upon execution further cause the benchmarking service to:
trigger execution of the execution plan on the execution resources.

17. The system of claim 15 , wherein the generation of the execution plan is further based on at least one of hardware available to a user requesting the benchmarking job, available jobs to batch with the task, previous execution of the task to be run as a job, and previous execution of similar jobs.

18. The system of claim 15 , wherein the generation of the execution plan includes executing the training of a model of the task to determine what hardware is capable of executing the task and meet the objective.

19. The system of claim 15 , wherein the execution of the task as a part of the benchmarking job according to the execution plan occurs in a virtual machine.

20. The system of claim 15 , wherein the execution of the task as a part of the benchmarking job according to the execution plan occurs in a container.

Assignees

• Amazon Tech Inc

Inventors

• Krishnamurthy Sandeep
• Chen Jiajie
• Esterhazy Jonathan
• Swamy Naveen Mysore Nagendra
• Yu Ruofei
• Wang Yao
• Nagmote Roshani
• Lupesko Hagay
• Madan Vikram

Classifications

• G06N20/00

  Machine learning · CPC title

• G06F11/3447 Primary

  Performance evaluation by modeling · CPC title

• G06F11/3452

  Performance evaluation by statistical analysis · CPC title

• G06F11/3428

  Benchmarking · CPC title

• G06F9/4881 Primary

  Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues · CPC title