Learning to select vocabularies for categorical features

Patent metadata

Field	Value
Publication number	US-11537664-B2
Application number	US-202016878912-A
Country	US
Kind code	B2
Filing date	May 20, 2020
Priority date	May 23, 2019
Publication date	Dec 27, 2022
Grant date	Dec 27, 2022

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for determining, for each of one or more categorical features, a respective vocabulary of categorical feature values of the categorical feature that should be active during processing of inputs by a machine learning model. In one aspect, a method comprises: generating a batch of output sequences, each output sequence in the batch specifying, for each of the categorical features, a respective vocabulary of categorical feature values of the categorical feature that should be active; for each output sequence in the batch, determining a performance metric of the machine learning model on a machine learning task after the machine learning model has been trained to perform the machine learning task with only the respective vocabulary of categorical feature values of each categorical feature specified by the output sequence being active.

First claim

What is claimed is:

1. A method performed by one or more data processing apparatus for determining, for each of one or more categorical features, a respective vocabulary of categorical feature values of the categorical feature that should be active during processing of inputs by a machine learning model having a plurality of machine learning model parameters, the method comprising:
generating, using a controller neural network having a plurality of controller parameters and in accordance with current values of the controller parameters, a batch of output sequences, each output sequence in the batch specifying, for each of the categorical features, a respective vocabulary of categorical feature values of the categorical feature that: (i) should be active during processing of inputs by the machine learning model, and (ii) is a subset of a discrete set of possible categorical feature values of the categorical feature;
for each output sequence in the batch, determining a performance metric of the machine learning model on a machine learning task after the machine learning model has been trained to perform the machine learning task with only the respective vocabulary of categorical feature values of each categorical feature specified by the output sequence being active, wherein during the training:
the machine learning model is configured to process an input that comprises one or more possible categorical feature values of respective categorical features by performing operations comprising:
for only those possible categorical feature values included in the input that are specified as active by the output sequence, mapping the possible categorical feature value to a corresponding embedding that is iteratively adjusted during the training; and
processing the embeddings to generate a machine learning model output for the input; and

using the performance metrics for the output sequences in the batch to adjust the current values of the controller parameters of the controller neural network.

2. The method of claim 1 , wherein:
each output sequence in the batch additionally specifies a respective embedding dimensionality corresponding to each categorical feature value that is specified by the output sequence as being active during processing of inputs by the machine learning model; and
for each output sequence in the batch, during the training the machine learning model is configured to map each possible categorical feature value included in the input that is specified as active by the output sequence to a corresponding embedding having the embedding dimensionality specified by the output sequence.

3. The method of claim 2 , wherein each output sequence specifies different embedding dimensionalities for different categorical feature values of a given categorical feature.

4. The method of claim 1 , wherein:
the machine learning model comprises a respective plurality of embedding models corresponding to each categorical feature;
an embedding model corresponding to a categorical feature defines a mapping of each of one or more categorical feature values of the categorical feature to a corresponding sub-embedding of the categorical feature value having a respective dimensionality;
each output sequence in the batch specifies, for each of the categorical features, one or more embedding models corresponding to the categorical feature; and
mapping a given categorical feature value of a given categorical feature included in the input to a corresponding embedding comprises:
for each embedding model specified by the output sequence for the given categorical feature that defines a mapping of the given categorical feature value to a corresponding sub-embedding, using the embedding model to map the given categorical feature value to the corresponding sub-embedding; and
determining the embedding of the given categorical feature value based on the sub-embeddings.

5. The method of claim 4 , wherein determining the embedding of the given categorical feature value based on the sub-embeddings comprises:
for any sub-embedding that has a dimensionality different than a specified dimensionality, projecting the sub-embedding to the specified dimensionality.

6. The method of claim 5 , wherein determining the embedding of the given categorical feature value based on the sub-embeddings further comprises summing or averaging the sub-embeddings.

7. The method of claim 1 , wherein processing the embeddings to generate a machine learning model output for the input comprises:
for each categorical feature, generating a combined embedding corresponding to the categorical feature based on the respective embedding of each categorical feature value of the categorical feature that is included in the input and that is specified as active by the output sequence; and
processing the combined embeddings using one or more neural network layers to generate the machine learning model output for the input.

8. The method of claim 1 , further comprising:
generating, using the controller neural network and in accordance with the adjusted values of the controller parameters, a new output sequence; and
training the machine learning model with only the respective vocabulary of categorical feature values of each categorical feature specified by the new output sequence being active on training data to determine adjusted values of the machine learning model parameters.

9. The method of claim 1 , wherein using the performance metrics for the output sequences in the batch to adjust the current values of the controller parameters of the controller neural network comprises:
training the controller neural network to generate output sequences that have increased performance metrics using a reinforcement learning technique.

10. The method of claim 9 , wherein the reinforcement learning technique is an actor critic reinforcement learning technique.

11. The method of claim 1 , wherein the performance metric for a given output sequence in the batch is based on a prediction accuracy of the machine learning model on the machine learning task with only the respective vocabulary of categorical feature values of each categorical feature specified by the output sequence being active.

12. The method of claim 2 , wherein the performance metric for a given output sequence in the batch is based on the respective embedding dimensionality corresponding to each categorical feature value that is specified by the output sequence as being active.

13. The method of claim 1 , wherein the controller neural network is a recurrent neural network that is configured to, for a given output sequence and at each of a plurality of time steps:
receive as input an representation of an output generated at the preceding time step in the given output sequence, and to process the input to update a current hidden state of the recurrent neural network and to generate a score distribution over possible outputs for the time step.

14. The method of claim 1 , further comprising:
generating, in accordance with the adjusted values of the controller parameters, a final output sequence that indicates, for each of the categorical features, a respective final vocabulary of categorical feature values.

15. The method of claim 14 , further comprising:
performing the machine learning task for received inputs by processing the received inputs with only the final vocabulary of categorical feature values of each categorical feature specified by the final output sequence being active.

16. The method of claim 1 , wherein the machine learning model is configured to process an input that characteriz

Assignees

• Google Llc

Inventors

• Li Cong
• Adams Jay
• Joglekar Manas
• Khaitan Pranav
• Le Quoc V
• Chen Mei

Classifications

• G06F2201/865

  Monitoring of software · CPC title

• G06N3/08

  Learning methods · CPC title

• G06F11/3409

  for performance assessment · CPC title

• G06F11/302

  where the computing system component is a software system · CPC title

• G06F11/3466

  Performance evaluation by tracing or monitoring · CPC title