Methods and apparatus for machine learning engine optimization

Patent metadata

Field	Value
Publication number	US-2021334585-A1
Application number	US-202016860930-A
Country	US
Kind code	A1
Filing date	Apr 28, 2020
Priority date	Apr 28, 2020
Publication date	Oct 28, 2021
Grant date	—

Abstract

Methods, apparatus, systems and articles of manufacture are disclosed for machine learning engine optimization. An example apparatus includes a selection metric analyzer to compute a first selection metric based on a first set of ordered output values from a first candidate machine learning engine and a set of reference data values; compute a second selection metric based on a second set of ordered output values from a second candidate machine learning engine and the set of reference data values; and a machine learning engine replacer to determine whether to replace an active machine learning engine with at least one of the first candidate machine learning engine or the second candidate machine learning engine based on the first selection metric and the second selection metric.

First claim

What is claimed is:

1 . An apparatus comprising:
a selection metric analyzer to: compute a first selection metric based on a first set of ordered output values from a first candidate machine learning engine and a set of reference data values; compute a second selection metric based on a second set of ordered output values from a second candidate machine learning engine and the set of reference data values; and a machine learning engine replacer to determine whether to replace an active machine learning engine with at least one of the first candidate machine learning engine or the second candidate machine learning engine based on the first selection metric and the second selection metric.

2 . The apparatus of claim 1 , wherein the first set of ordered output values are ranked by order of importance based on a configuration of the first candidate machine learning engine, and the second set of ordered output values are ranked by order of importance based on a configuration of the second candidate machine learning engine.

3 . The apparatus of claim 2 , wherein the set of reference data values includes unordered reference data values that are utilized to determine consecutive highest rankings in the first set of ordered output values from the first machine learning engine and the second set of ordered output values from second first machine learning engine.

4 . The apparatus of claim 3 , wherein the determining of the first selection metric further includes the selection metric analyzer to determine a first evaluation metric by:
determining a first difference between 1) a number of the unordered reference data values in the set of reference data values and 2) a number of the unordered reference data values included in beginning positions of the first set of ordered output values from the first candidate machine learning engine; and dividing the first difference by the number of the unordered reference data values from the set of reference data values.

5 . The apparatus of claim 4 , wherein the determining of the first selection metric further includes the selection metric analyzer to determine a second evaluation metric by:
determining a second difference between 1) a number corresponding to how many of the unordered reference data values from the set of reference data values are present in the first set of ordered output values and 2) the number of the unordered reference data values from the set of reference data values; adding a penalty to the second difference for each of the unordered reference data values from the set of reference data values that is not present in the first set of ordered output values; dividing the second difference by an addition of 1) the number of values in the first set of ordered output values and 2) the number of the unordered reference data values from the set of reference data values.

6 . The apparatus of claim 5 , wherein the penalty has a value of one-fourth.

7 . The apparatus of claim 5 , wherein the determining the first selection metric further includes the selection metric analyzer to:
add the first evaluation metric and the second evaluation metric to determine a result; and divide the result by two to determine the first selection metric.

8 . The apparatus of claim 1 , wherein determining whether to replace the active machine learning engine with at least one of the first candidate machine learning engine or the second candidate machine learning engine further includes the selection metric analyzer to:
compare the first selection metric to the second selection metric to identify the lowest selection metric; compare the lowest selection metric to a threshold to determine if the at least one of the first candidate machine learning engine or the second candidate machine learning should replace the active machine learning engine; and replace the active machine learning engine with the at least one of the first candidate machine learning engine or the second candidate machine learning when the lowest selection metric satisfies the threshold.

9 . A non-transitory computer readable storage medium comprising instructions which, when executed, cause one or more processors to at least:
compute a first selection metric based on a first set of ordered output values from a first candidate machine learning engine and a set of reference data values; compute a second selection metric based on a second set of ordered output values from a second candidate machine learning engine and the set of reference data values; and determine whether to replace an active machine learning engine with at least one of the first candidate machine learning engine or the second candidate machine learning engine based on the first selection metric and the second selection metric.

10 . The computer readable storage medium of claim 9 , wherein the first set of ordered output values are ranked by order of importance based on a configuration of the first candidate machine learning engine, and the second set of ordered output values are ranked by order of importance based on a configuration of the second candidate machine learning engine.

11 . The computer readable storage medium of claim 10 , wherein the set of reference data values includes unordered reference data values that are utilized to determine consecutive highest rankings in the first set of ordered output values from the first machine learning engine and the second set of ordered output values from second first machine learning engine.

12 . The computer readable storage medium of claim 11 , wherein the instructions, when executed, cause the one or more processors to determine a first evaluation metric by:
determining a first difference between 1) a number of the unordered reference data values in the set of reference data values and 2) a number of the unordered reference data values included in beginning positions of the first set of ordered output values from the first candidate machine learning engine; and dividing the first difference by the number of the unordered reference data values from the set of reference data values.

13 . The computer readable storage medium of claim 12 , wherein the instructions, when executed, cause the one or more processors to determine a second evaluation metric by:
determining a second difference between 1) a number corresponding to how many of the unordered reference data values from the set of reference data values are present in the first set of ordered output values and 2) the number of the unordered reference data values from the set of reference data values; adding a penalty to the second difference for each of the unordered reference data values from the set of reference data values that is not present in the first set of ordered output values; dividing the second difference by an addition of 1) the number of values in the first set of ordered output values and 2) the number of the unordered reference data values from the set of reference data values.

14 . The computer readable storage medium of claim 13 , wherein the instructions, when executed, cause the one or more processors to:
add the first evaluation metric and the second evaluation metric to determine a result; and divide the result by two to determine the first selection metric.

15 . A method comprising:
computing, by executing an instruction with a processor, a first selection metric based on a first set of ordered output values from a first candidate machine learning engine and a set of reference data values; computing, by executing an instruction with the processor, a second selection metric based on a

Assignees

• Nielsen Co Us Llc

Inventors

• Szabo Rachel Anne

Classifications

• G06F18/285

  Selection of pattern recognition techniques, e.g. of classifiers in a multi-classifier system · CPC title

• G06N20/20 Primary

  Ensemble learning · CPC title

• G06F18/217

  Validation; Performance evaluation; Active pattern learning techniques · CPC title

• G06F18/2113

  by ranking or filtering the set of features, e.g. using a measure of variance or of feature cross-correlation · CPC title

• G06V10/751

  Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching · CPC title