System and method for ensemble expert diversification via bidding and control thereof

Patent metadata

Field	Value
Publication number	US-12141237-B2
Application number	US-202016944503-A
Country	US
Kind code	B2
Filing date	Jul 31, 2020
Priority date	Jul 31, 2020
Publication date	Nov 12, 2024
Grant date	Nov 12, 2024

Abstract

The present teaching relates to method, system, medium, and implementations for machine learning. A bid is received, from an expert during training, for a training sample with an amount within a level of available bidding currency associated with the expert. The training sample is used for training a model associated with the expert. It is determined whether the expert is among at least one winner selected based on bids from one or more experts. If the expert is among the at least one winner, the training sample is sent to the expert. The at least one winner is selected based on one or more criteria aiming at expert diversification.

First claim

We claim:

1. A method implemented on at least one machine including at least one processor, memory, and communication platform capable of connecting to a network for machine learning, the method comprising:
allocating, by a training controller to each of multiple experts, a limited level of available bidding currency;
receiving, by the training controller from each of the multiple experts during training, a bid for a training sample with an amount within the limited level of available bidding currency associated with the expert;
determining, by the training controller based on the amounts, a sub-group of the multiple experts to receive the training sample;
sending, by the training controller, the training sample to the sub-group;
receiving, by the training controller from each expert in the subgroup, a prediction generated based on the training sample and one or more parameters associated with a model trained by the expert;
sending, by the training controller to at least one expert in the sub-group based on a confidence level associated with the prediction made based on each model associated with each expert in the sub-group, a ground truth label corresponding to the training sample to enable the at least one expert to train the model based on the ground truth label; and
adjusting, by the training controller based on the confidence level associated with the prediction made by each expert in the sub-group, the limited level of available bidding currency associated with one or more experts in the sub-group.

2. The method of claim 1 , further comprising initializing the limited level of available bidding currency associated with each expert.

3. The method of claim 1 , further comprising receiving, from each expert, a metric characterizing a corresponding prediction, wherein the metric is used to determine whether the ground truth label for the training sample is to be provided to the at least one expert in the sub-group to facilitate the training.

4. The method of claim 3 , wherein the metric includes a confidence score indicative of the confidence level that each expert has in its prediction.

5. The method of claim 3 , wherein the adjusting the limited level of available bidding currency is based on the metric and the amount.

6. Machine readable and non-transitory medium having information recorded thereon for machine learning, wherein the information, once read by the machine, causes the machine to perform:
allocating, by a training controller to each of multiple experts, a limited level of available bidding currency;
receiving, by the training controller from each of the multiple experts during training, a bid for a training sample with an amount within the limited level of available bidding currency associated with the expert;
determining, by the training controller based on the amounts, a sub-group of the multiple experts to receive the training sample;
sending, by the training controller, the training sample to the sub-group;
receiving, by the training controller from each expert in the subgroup, a prediction generated based on the training sample and one or more parameters associated with a model trained by the expert;
sending, by the training controller to at least one expert in the sub-group based on a confidence level associated with the prediction made based on each model associated with each expert in the sub-group, a ground truth label corresponding to the training sample to enable the at least one expert to train the model based on the ground truth label; and
adjusting, by the training controller based on the confidence level associated with the prediction made by each expert in the sub-group, the limited level of available bidding currency associated with one or more experts in the sub-group.

7. The medium of claim 6 , wherein the information, once read by the machine, further causes the machine to perform initializing the limited level of available bidding currency associated with each expert.

8. The medium of claim 6 , wherein the information, once read by the machine, further causes the machine to perform receiving, from each expert, a metric characterizing a corresponding prediction, wherein the metric is used to determine whether the ground truth label for the training sample is to be provided to the at least one expert in the sub-group to facilitate the training.

9. The medium of claim 8 , wherein the metric includes a confidence score indicative of the confidence level that each expert has in its prediction.

10. The medium of claim 8 , wherein the information, once read by the machine, further causes the machine to perform adjusting the limited level of available bidding currency based on the metric and the amount.

11. A system for machine learning, comprising:
a currency allocation updater configured for
allocating, to each of multiple experts, a limited level of available bidding currency;
a bidding winner selector implemented by a processor and configured for
receiving, from each of the multiple experts during training, a bid for a training sample with an amount within the limited level of available bidding currency associated with the expert, and
determining, based on the amounts, a sub-group of the multiple experts to receive the training sample;
a training data distribution unit implemented by the processor and configured for sending the training sample to the sub-group; and
a ground truth allocation unit configured for
receiving, from each expert in the subgroup, a prediction generated based on the training sample and one or more parameters associated with a model trained by the expert, and
sending, to at least one expert in the sub-group based on a confidence level associated with the prediction made based on each model associated with each expert in the sub-group, a ground truth label corresponding to the training sample to enable the at least one expert to train the model based on the ground truth label,
wherein the currency allocation updater is further configured for
adjusting, based on the confidence level associated with the prediction made by each expert in the sub-group, the limited level of available bidding currency associated with one or more experts in the sub-group.

12. The system of claim 11 , further comprising an initialization unit configured for initializing the limited level of available bidding currency associated with each expert.

13. The system of claim 11 , wherein the ground truth allocation unit is further configured for receiving, from each expert, a metric characterizing a corresponding prediction, wherein the metric is used to determine whether the ground truth label for the training sample is to be provided to the at least one expert in the sub-group to facilitate the training, wherein the metric includes a confidence score indicative of a confidence level that each expert has in its prediction.

14. The system of claim 13 , wherein the currency allocation updater is further configured for updating the limited level of available bidding currency based on the metric and the amount.

Assignees

• Yahoo Assets Llc

Inventors

• Lalouche Gal
• Wolff Ran

Classifications

• G06F18/2155

  characterised by the incorporation of unlabelled data, e.g. multiple instance learning [MIL], semi-supervised techniques using expectation-maximisation [EM] or naïve labelling · CPC title

• G06N20/20 Primary

  Ensemble learning · CPC title

• G06Q30/0241

  Advertisements · CPC title

• G06F18/2185 Primary

  the supervisor being an automated module, e.g. intelligent oracle · CPC title