Systems and methods for Bayesian optimization using integrated acquisition functions

Patent metadata

Field	Value
Publication number	US-9864953-B2
Application number	US-201414291212-A
Country	US
Kind code	B2
Filing date	May 30, 2014
Priority date	May 30, 2013
Publication date	Jan 9, 2018
Grant date	Jan 9, 2018

Abstract

Techniques for use in connection with performing optimization using an objective function. The techniques include using at least one computer hardware processor to perform: identifying, using an integrated acquisition utility function and a probabilistic model of the objective function, at least a first point at which to evaluate the objective function; evaluating the objective function at least at the identified first point; and updating the probabilistic model of the objective function using results of the evaluating to obtain an updated probabilistic model of the objective function.

First claim

What is claimed is:

1. A system for optimizing performance of a machine learning system, the system comprising:
at least one computer hardware processor; and
at least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by the at least one computer hardware processor, cause the at least one computer hardware processor to perform:
identifying at least a first set of hyper-parameter values at which to evaluate an objective function that relates hyper-parameter values of the machine learning system to values providing a measure of performance of the machine learning system, the identifying performed at least in part by using an integrated acquisition utility function and a probabilistic model of the objective function, wherein the integrated acquisition utility function is obtained at least in part by integrating an initial acquisition utility function with respect to at least one parameter of the probabilistic model;
evaluating the objective function at least at the identified first set of hyper-parameter values, at least in part by executing the machine learning system when configured with the first set of hyper-parameter values, to obtain a corresponding first value providing a measure of performance of the machine learning system when operated using the first set of hyper-parameter values; and
updating the probabilistic model of the objective function using results of the evaluating to obtain an updated probabilistic model of the objective function.

2. The system of claim 1 , wherein the objective function relates values of a plurality of hyper-parameters of a neural network for identifying objects in images to respective values providing a measure of performance of the neural network in identifying the objects in the images.

3. The system of claim 1 , wherein the processor-executable instructions further cause the at least one computer hardware processor to perform:
identifying, using the integrated acquisition utility function and the updated probabilistic model of the objective function, at least a second set of hyper-parameter values point at which to evaluate the objective function; and
evaluating the objective function at least at the identified second set of hyper-parameter values, at least in part by executing the machine learning system when configured with the first set of hyper-parameter values, to obtain a corresponding second value providing a measure of performance of the machine learning system when operated using the second set of hyper-parameter values.

4. The system of claim 1 , wherein the initial acquisition utility function is comprises an acquisition utility function selected from a group consisting of: a probability of improvement utility function, an expected improvement utility function, a regret minimization utility function, and an entropy-based utility function.

5. The system of claim 1 , wherein the probabilistic model of the objective function comprises a Gaussian process.

6. The system of claim 1 , wherein the identifying is performed at least in part by using a Markov chain Monte Carlo technique.

7. The system of claim 1 , wherein the processor-executable instructions further cause the at least one computer hardware processor to perform:
identifying a plurality of sets of hyper-parameter values at which to evaluate the objective function;
evaluating the objective function at each of the plurality of sets of hyper-parameter values; and
identifying or approximating, based on results of the evaluating, a set of hyper-parameter values at which the objective function attains a maximum value.

8. The system of claim 1 , wherein the probabilistic model of the objective function comprises a neural network.

9. A method for optimizing performance of a machine learning system, the method comprising:
using at least one computer hardware processor to perform:
identifying at least a first set of hyper-parameter values at which to evaluate an objective function that relates hyper-parameter values of the machine learning system to values providing a measure of performance of the machine learning system, the identifying performed at least in part by using an integrated acquisition utility function and a probabilistic model of the objective function, wherein the integrated acquisition utility function is obtained at least in part by integrating an initial acquisition utility function with respect to at least one parameter of the probabilistic model;
evaluating the objective function at least at the identified first set of hyper-parameter values, at least in part by executing the machine learning system when configured with the first set of hyper-parameter values, to obtain a corresponding first value providing a measure of performance of the machine learning system when operated using the first set of hyper-parameter values; and
updating the probabilistic model of the objective function using results of the evaluating to obtain an updated probabilistic model of the objective function.

10. The method of claim 9 , wherein the processor-executable instructions further cause the at least one computer hardware processor to perform:
identifying, using the integrated acquisition utility function and the updated probabilistic model of the objective function, at least a second set of hyper-parameter values at which to evaluate the objective function; and
evaluating the objective function at least at the identified second set of hyper-parameter values, at least in part by executing the machine learning system when configured with the first set of hyper-parameter values, to obtain a corresponding second value providing a measure of performance of the machine learning system when operated using the second set of hyper-parameter values.

11. The method of claim 9 , wherein the probabilistic model of the objective function comprises a Gaussian process.

12. The method of claim 9 , wherein the identifying is performed at least in part by using a Markov chain Monte Carlo technique.

13. The method of claim 9 , wherein the processor-executable instructions further cause the at least one computer hardware processor to perform:
identifying a plurality of sets of hyper-parameter values at which to evaluate the objective function;
evaluating the objective function at each of the plurality of sets of hyper-parameter values; and
identifying or approximating, based on results of the evaluating, a set of hyper-parameter values at which the objective function attains a maximum value.

14. The method of claim 9 , wherein the probabilistic model of the objective function comprises a neural network.

15. At least one non-transitory computer-readable storage medium storing processor executable instructions that, when executed by at least one computer hardware processor, cause the at least one computer hardware processor to perform a method for optimizing a machine learning system, the method comprising:
identifying at least a first set of hyper-parameter values at which to evaluate an objective function that relates hyper-parameter values of the machine learning system to values providing a measure of performance of the machine learning system, the identifying performed at least in part by using an integrated acquisition utility function and a probabilistic model of the objective function, wherein the integrated acquisition utility function is obtained at least in part by integrating an initial acquisition utility function with respect to at least one parameter of the probabilistic model;
evaluating the objective function at least at the identified first set of

Assignees

• Univ Sherbrooke
• Harvard College
• Governing Council Of The Univ Of Toronto The
• Socpra Sciences Et Genie Sec

Inventors

• Adams Ryan P
• Snoek Roland Jasper
• Larochelle Hugo

Classifications

• G06N7/01 Primary

  Probabilistic graphical models, e.g. probabilistic networks · CPC title

• G06N20/10 Primary

  using kernel methods, e.g. support vector machines [SVM] · CPC title

• G06N5/048 Primary

  Fuzzy inferencing · CPC title

• G06N20/00

  Machine learning · CPC title

• G06F17/11

  for solving equations {, e.g. nonlinear equations, general mathematical optimization problems (optimization specially adapted for a specific administrative, business or logistic context G06Q10/04)} · CPC title