Method, apparatus, and system for selecting training observations for machine learning models

Patent metadata

Field	Value
Publication number	US-2019102674-A1
Application number	US-201715721002-A
Country	US
Kind code	A1
Filing date	Sep 29, 2017
Priority date	Sep 29, 2017
Publication date	Apr 4, 2019
Grant date	—

Abstract

An approach is provided for selecting training observations for machine learning models. The approach involves determining a first distribution of a plurality of features observed in the training data set, and a second distribution of the plurality of features observed in the candidate pool of observations. The approach further involves selecting one or more observations in the candidate pool of observations for annotation based on the first distribution and the second distribution. The approach further involves adding the one or more observations to the training data set after annotation. The training data set is used for training the machine learning model.

First claim

What is claimed is:

1 . A computer-implemented method for sampling from a candidate pool of observations to create a training data set for a machine learning model comprising:
determining, by a processor, a first distribution of a plurality of features observed in the training data set; determining a second distribution of the plurality of features observed in the candidate pool of observations; selecting one or more observations in the candidate pool of observations for annotation based on the first distribution and the second distribution; and adding the one or more observations to the training data set after annotation, wherein the training data set is used for training the machine learning model.

2 . The method of claim 1 , wherein a sampling probability of the one or more selected observations is based on a similarity of the one or more observations to other observations in the training data set and the candidate pool of observations.

3 . The method of claim 1 , further comprising:
determining a sampling probability for the one or more observations based on a product of the first distribution and the second distribution, wherein the one or more observations are selected from the candidate pool of observations based on the sampling probability.

4 . The method of claim 1 , wherein the plurality of features includes an individual observation of the training data set, metadata describing the training observations, characteristics derived from the observations, or a combination thereof.

5 . The method of claim 4 , wherein the metadata describing the training observations include a geographic location where a respective one of the training observations was collected, map features associated with the geographic location, or a combination thereof.

6 . The method of claim 1 , further comprising:
creating a feature space for each observation of the candidate pool of observations based on the plurality of features associated with said each observation; and calculating a score for said each observation based on the plurality of features, wherein the one or more observations are selected based on the score for said each observation.

7 . The method of claim 6 , further comprising:
determining a distribution of the score for said each observation, wherein the one or more observations are further based on the distribution.

8 . The method of claim 6 , wherein the score indicates whether said each observation is an outlier or an inlier with respect to the feature space.

9 . The method of claim 1 , wherein the one or more observations are selected to be added to the training data (a) when the training data set and the candidate pool of observations are first created, (b) at a fixed frequency, (c) as the candidate pool of observations is collected, or (d) a combination thereof.

10 . The method of claim 1 , further comprising:
iteratively determining the first distribution and the second distribution as the one or more observations are selected to be added to the training data set.

11 . An apparatus for sampling from a candidate pool of observations to create a training data set for a machine learning model comprising:
at least one processor; and at least one memory including computer program code for one or more programs, the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus to perform at least the following,
determine a first distribution of a plurality of features observed in the training data set;
determine a second distribution of the plurality of features observed in the candidate pool of observations;
select one or more observations in the candidate pool of observations for annotation based on the first distribution and the second distribution; and
add the one or more selected observations to the training data set after annotation,
wherein the training data set is used for training the machine learning model.

12 . The apparatus of claim 11 , wherein a sampling probability of the one or more selected observations is based on a similarity of the one or more observations to other observations in the training data set and the candidate pool of observations.

13 . The apparatus of claim 11 , wherein the apparatus is further caused to:
determine a sampling probability for the one or more observations based on a product of the first distribution and the second distribution, wherein the one or more observations are selected from the candidate pool of observations based on the sampling probability.

14 . The apparatus of claim 11 , wherein the plurality of features includes an individual observation of the training data set, metadata describing the training observations, characteristics derived from the observations, or a combination thereof.

15 . The apparatus of claim 14 , wherein the metadata describing the training observations include a geographic location where a respective one of the training observations was collected, map features associated with the geographic location, or a combination thereof.

16 . A non-transitory computer-readable storage medium for sampling from a candidate pool of observations to create a training data set for a machine learning model, carrying one or more sequences of one or more instructions which, when executed by one or more processors, cause an apparatus to perform:
determining, by a processor, a first distribution of a plurality of features observed in the training data set; determining a second distribution of the plurality of features observed in the candidate pool of observations; selecting one or more observations in the candidate pool of observations for annotation based on the first distribution and the second distribution; and adding the one or more selected observations to the training data set after annotation, wherein the training data set is used for training the machine learning model.

17 . The non-transitory computer-readable storage medium of claim 16 , wherein the apparatus further is caused to perform:
creating a feature space for each observation of the candidate pool of observations based on the plurality of features associated with said each observation; and calculating a score for said each observation based on the plurality of features, wherein the one or more observations are selected based on the score for said each observation.

18 . The non-transitory computer-readable storage medium of claim 17 , wherein the apparatus further is caused to perform:
determining a distribution of the score for said each observation, wherein the one or more observations are further based on the distribution.

19 . The non-transitory computer-readable storage medium of claim 17 , wherein the score indicates whether said each observation is an outlier or an inlier with respect to the feature space.

20 . The non-transitory computer-readable storage medium of claim 17 , further comprising:
iteratively determining the first distribution and the second distribution as the one or more observations are selected to be added to the training data set.

Assignees

• Here Global Bv

Inventors

• Kwant Richard
• Mittal Anish
• Lawlor David

Classifications

• G06N3/044

  Recurrent networks, e.g. Hopfield networks · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06N20/10 Primary

  using kernel methods, e.g. support vector machines [SVM] · CPC title

• G06F16/35

  Clustering; Classification · CPC title

• G06F16/93

  Document management systems · CPC title