Systems and methods for Bayesian optimization using non-linear mapping of input

Patent metadata

Field	Value
Publication number	US-10074054-B2
Application number	US-201414291379-A
Country	US
Kind code	B2
Filing date	May 30, 2014
Priority date	May 30, 2013
Publication date	Sep 11, 2018
Grant date	Sep 11, 2018

Abstract

Techniques for use in connection with performing optimization using an objective function that maps elements in a first domain to values in a range. The techniques include using at least one computer hardware processor to perform: identifying a first point at which to evaluate the objective function at least in part by using an acquisition utility function and a probabilistic model of the objective function, wherein the probabilistic model depends on a non-linear one-to-one mapping of elements in the first domain to elements in a second domain; evaluating the objective function at the identified first point to obtain a corresponding first value of the objective function; and updating the probabilistic model of the objective function using the first value to obtain an updated probabilistic model of the objective function.

First claim

What is claimed is:

1. A system for optimizing performance of a machine learning system, the system comprising:
at least one computer hardware processor; and
at least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by the at least one computer hardware processor, cause the at least one computer hardware processor to perform:
identifying a first set of hyper-parameter values at which to evaluate an objective function relating values of hyper-parameters of the machine learning system to values providing measures of performance of the machine learning system, the identifying performed at least in part by using an acquisition utility function and a probabilistic model of the objective function, the probabilistic model of the objective function comprising a stationary probabilistic model composed with a non-linear bijective one-to-one mapping of the values of the hyper-parameters from a first domain to a second domain;
evaluating the objective function at the identified first set of hyper-parameter values, at least in part by executing the machine learning system when configured with the first set of hyper-parameter values, to obtain a corresponding first value providing a measure of performance of the machine learning system when operated using the first set of hyper-parameter values; and
updating the probabilistic model of the objective function using the first value to obtain an updated probabilistic model of the objective function.

2. The system of claim 1 , wherein the objective function relates values of a plurality of hyper-parameters of a neural network for identifying objects in images to respective values providing a measure of performance of the neural network in identifying the objects in the images.

3. The system of claim 1 , wherein the processor-executable instructions further cause the at least one computer hardware processor to perform:
identifying a second set of hyper-parameter values at which to evaluate the objective function;
evaluating the objective function at the identified second set of hyper-parameter values, at least in part by executing the machine learning system when configured with the second set of hyper-parameter values, to obtain a corresponding second value providing a measure of performance of the machine learning system when operated using the second set of hyper-parameter values; and
updating the updated probabilistic model of the objective function using the second value to obtain a second updated probabilistic model of the objective function.

4. The system of claim 1 , wherein the non-linear one-to-one mapping comprises a cumulative distribution function of a Beta distribution.

5. The system of claim 1 , wherein the acquisition utility function is an integrated acquisition utility function.

6. The system of claim 1 , wherein the stationary probabilistic model comprises a Gaussian process.

7. The system of claim 1 , wherein evaluating the objective function at the identified first set of hyper-parameter values comprises:
configuring the machine learning system by setting the hyper-parameters of the machine learning system to the first set of hyper-parameter values; and operating the configured machine learning system to obtain the first value.

8. A method optimizing performance of a machine learning system, the method comprising:
using at least one computer hardware processor to perform:
identifying a first set of hyper-parameter values at which to evaluate an objective function relating values of hyper-parameters of the machine learning system to values providing measures of performance of the machine learning system, the identifying performed at least in part by using an acquisition utility function and a probabilistic model of the objective function, the probabilistic model of the objective function comprising a stationary probabilistic model composed with a non-linear bijective one-to-one mapping of the values of the hyper-parameters from a first domain to a second domain;
evaluating the objective function at the identified first set of hyper-parameter values, at least in part by executing the machine learning system when configured with the first set of hyper-parameter values, to obtain a corresponding first value providing a measure of performance of the machine learning system when operated using the first set of hyper-parameter values; and
updating the probabilistic model of the objective function using the first value to obtain an updated probabilistic model of the objective function.

9. The method of claim 8 , wherein the objective function relates values of a plurality of hyper-parameters of a neural network for identifying objects in images to respective values providing a measure of performance of the neural network in identifying the objects in the images.

10. The method of claim 8 , wherein the processor-executable instructions further cause the at least one computer hardware processor to perform:
identifying a second set of hyper-parameter values at which to evaluate the objective function;
evaluating the objective function at the identified second set of hyper-parameter values, at least in part by executing the machine learning system when configured with the second set of hyper-parameter values, to obtain a corresponding second value providing a measure of performance of the machine learning system when operated using the second set of hyper-parameter values; and
updating the updated probabilistic model of the objective function using the second value to obtain a second updated probabilistic model of the objective function.

11. The method of claim 8 , wherein the non-linear one-to-one mapping comprises a cumulative distribution function of a Beta distribution.

12. The method of claim 11 , wherein the acquisition utility function is an integrated acquisition utility function.

13. The method of claim 8 , wherein evaluating the objective function at the identified first set of hyper-parameter values comprises:
configuring the machine learning system by setting the hyper-parameters of the machine learning system to the first set of hyper-parameter values; and operating the configured machine learning system to obtain the first value.

14. At least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by the at least one computer hardware processor, cause the at least one computer hardware processor to perform a method for optimizing performance of a machine learning system, the method comprising:
identifying a first set of hyper-parameter values at which to evaluate an objective function relating values of hyper-parameters of the machine learning system to values providing measures of performance of the machine learning system, the identifying performed at least in part by using an acquisition utility function and a probabilistic model of the objective function, the probabilistic model of the objective function comprising a stationary probabilistic model composed with a non-linear bijective one-to-one mapping of the values of the hyper-parameters from a first domain to a second domain; and
evaluating the objective function at the identified first set of hyper-parameter values, at least in part by executing the machine learning system when configured with the first set of hyper-parameter values, to obtain a corresponding first value providing a measure of performance of the machine learning system when operated using the first set of hyper-parameter values.

15. The at least one non-transitory computer-readable storage medium of clai

Assignees

• Harvard College
• Governing Council Univ Toronto
• Governing Council Of The Univ Of Toronto The

Inventors

• Adams Ryan P
• Snoek Roland Jasper
• Swersky Kevin
• Zemel Richard

Classifications

• G06N7/01 Primary

  Probabilistic graphical models, e.g. probabilistic networks · CPC title

• G06N20/10 Primary

  using kernel methods, e.g. support vector machines [SVM] · CPC title

• G06N5/048 Primary

  Fuzzy inferencing · CPC title

• G06N20/00

  Machine learning · CPC title

• G06F17/11

  for solving equations {, e.g. nonlinear equations, general mathematical optimization problems (optimization specially adapted for a specific administrative, business or logistic context G06Q10/04)} · CPC title