Learning to select vocabularies for categorical features

Patent metadata

Field	Value
Publication number	US-11714857-B2
Application number	US-202218076662-A
Country	US
Kind code	B2
Filing date	Dec 7, 2022
Priority date	May 23, 2019
Publication date	Aug 1, 2023
Grant date	Aug 1, 2023

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for determining, for each of one or more categorical features, a respective vocabulary of categorical feature values of the categorical feature that should be active during processing of inputs by a machine learning model. In one aspect, a method comprises: generating a batch of output sequences, each output sequence in the batch specifying, for each of the categorical features, a respective vocabulary of categorical feature values of the categorical feature that should be active; for each output sequence in the batch, determining a performance metric of the machine learning model on a machine learning task after the machine learning model has been trained to perform the machine learning task with only the respective vocabulary of categorical feature values of each categorical feature specified by the output sequence being active.

First claim

What is claimed is:

1. A method performed by one or more data processing apparatus for determining, for each of one or more categorical feature values, a respective embedding dimensionality of an embedding that should be generated for the categorical feature value during processing of inputs by a machine learning model having a plurality of machine learning model parameters, the method comprising:
generating, using a controller neural network having a plurality of controller parameters and in accordance with current values of the controller parameters, a batch of output sequences, each output sequence in the batch specifying, for each categorical feature value, a respective embedding dimensionality of an embedding that should be generated for the categorical feature value during processing of inputs by the machine learning model;
for each output sequence in the batch, determining a performance metric of the machine learning model on a machine learning task when the machine learning model generates embeddings of the categorical feature values with the embedding dimensionalities specified by the output sequence, wherein:
the machine learning model is configured to process an input that comprises one or more categorical feature values by performing operations comprising:
mapping each categorical feature value included in the input to an embedding having the corresponding embedding dimensionality specified by the output sequence; and
processing the embeddings to generate a machine learning model output for the input; and

using the performance metrics for the output sequences in the batch to adjust the current values of the controller parameters of the controller neural network.

2. The method of claim 1 , wherein each output sequence specifies different embedding dimensionalities for different categorical feature values of a given categorical feature.

3. The method of claim 1 , wherein:
the machine learning model comprises a respective plurality of embedding models corresponding to each categorical feature;
an embedding model corresponding to a categorical feature defines a mapping of each of one or more categorical feature values of the categorical feature to a corresponding sub-embedding of the categorical feature value having a respective dimensionality;
each output sequence in the batch specifies, for each of the categorical features, one or more embedding models corresponding to the categorical feature; and
mapping a given categorical feature value of a given categorical feature included in the input to an embedding comprises:
for each embedding model specified by the output sequence for the given categorical feature that defines a mapping of the given categorical feature value to a corresponding sub-embedding, using the embedding model to map the given categorical feature value to the corresponding sub-embedding; and
determining the embedding of the given categorical feature value based on the sub-embeddings.

4. The method of claim 3 , wherein determining the embedding of the given categorical feature value based on the sub-embeddings comprises:
for any sub-embedding that has a dimensionality different than a specified dimensionality, projecting the sub-embedding to the specified dimensionality.

5. The method of claim 4 , wherein determining the embedding of the given categorical feature value based on the sub-embeddings further comprises summing or averaging the sub-embeddings.

6. The method of claim 1 , wherein processing the embeddings to generate a machine learning model output for the input comprises:
for each categorical feature, generating a combined embedding corresponding to the categorical feature based on the respective embedding of each categorical feature value of the categorical feature that is included in the input; and
processing the combined embeddings using one or more neural network layers to generate the machine learning model output for the input.

7. The method of claim 1 , further comprising:
generating, using the controller neural network and in accordance with the adjusted values of the controller parameters, a new output sequence; and
training the machine learning model with the machine learning model being configured to generate embeddings of categorical feature values with embedding dimensionalities specified by the new output sequence to determine adjusted values of the machine learning model parameters.

8. The method of claim 1 , wherein using the performance metrics for the output sequences in the batch to adjust the current values of the controller parameters of the controller neural network comprises:
training the controller neural network to generate output sequences that have increased performance metrics using a reinforcement learning technique.

9. The method of claim 8 , wherein the reinforcement learning technique is an actor critic reinforcement learning technique.

10. The method of claim 1 , wherein the performance metric for a given output sequence in the batch is based on a prediction accuracy of the machine learning model on the machine learning task when the machine learning model is configured to generate embeddings of categorical feature values with embedding dimensionalities specified by the given output sequence.

11. The method of claim 1 , wherein the performance metric for a given output sequence in the batch is based on the respective embedding dimensionality corresponding to each categorical feature value.

12. The method of claim 1 , wherein the controller neural network is a recurrent neural network that is configured to, for a given output sequence and at each of a plurality of time steps:
receive as input an representation of an output generated at a preceding time step in the given output sequence, and to process the input to update a current hidden state of the recurrent neural network and to generate a score distribution over possible outputs for the time step.

13. The method of claim 1 , further comprising:
generating, in accordance with the adjusted values of the controller parameters, a final output sequence that indicates, for each categorical feature value, a respective embedding dimensionality of the embedding that should be generated for the categorical feature value during processing of inputs by the machine learning model.

14. The method of claim 13 , further comprising:
performing the machine learning task for received inputs by processing the received inputs using the machine learning model when the machine learning model is configured to generate a respective embedding for each categorical feature value that has the dimensionality specified by the final output sequence.

15. The method of claim 1 , wherein the machine learning model is configured to process an input that characterizes previous search queries of a user to generate an output that characterizes a predicted next search query of the user.

16. The method of claim 1 , wherein the machine learning model is configured to process an input that characterizes previous videos watched by a user to generate an output that characterizes a predicted next video watched by the user.

17. The method of claim 1 , wherein:
each output sequence in the batch additionally specifies an architecture of the machine learning model; and
for each output sequence in the batch, the machine learning model is configured to process the input using the machine learning model architecture specified by the output sequence.

18. The method of claim 1 , wherein:
each output sequence in the batch additionally specifies, for each of on

Assignees

• Google Llc

Inventors

• Li Cong
• Adams Jay
• Joglekar Manas
• Khaitan Pranav
• Le Quoc V
• Chen Mei

Classifications

• G06N3/092

  Reinforcement learning · CPC title

• G06N3/0985

  Hyperparameter optimisation; Meta-learning; Learning-to-learn · CPC title

• G06N3/082

  modifying the architecture, e.g. adding, deleting or silencing nodes or connections · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/0442

  characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] · CPC title