On-device neural networks for natural language understanding

Patent metadata

Field	Value
Publication number	US-10885277-B2
Application number	US-201816135545-A
Country	US
Kind code	B2
Filing date	Sep 19, 2018
Priority date	Aug 2, 2018
Publication date	Jan 5, 2021
Grant date	Jan 5, 2021

Abstract

The present disclosure provides projection neural networks and example applications thereof. In particular, the present disclosure provides a number of different architectures for projection neural networks, including two example architectures which can be referred to as: Self-Governing Neural Networks (SGNNs) and Projection Sequence Networks (ProSeqoNets). Each projection neural network can include one or more projection layers that project an input into a different space. For example, each projection layer can use a set of projection functions to project the input into a bit-space, thereby greatly reducing the dimensionality of the input and enabling computation with lower resource usage. As such, the projection neural networks provided herein are highly useful for on-device inference in resource-constrained devices. For example, the provided SGNN and ProSeqoNet architectures are particularly beneficial for on-device inference such as, for example, solving natural language understanding tasks on-device.

First claim

What is claimed is:

1. A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to implement:
a projection neural network configured to receive a projection network input and to generate a projection network output from the projection network input, the projection neural network comprising:
a sequence of one or more projection layers, wherein each projection layer has a plurality of projection layer parameters that were learned during training of the projection neural network, wherein each projection layer is configured to:
receive a layer input;
apply a plurality of projection layer functions to the layer input, each projection layer function generating a respective projection function output that projects the layer input to a different space, wherein the plurality of projection layer functions comprise a plurality of locality sensitive hashing functions that were held static during training of the projection neural network, and
generate a layer output by applying the projection layer parameters for the projection ayer to the projection function outputs; and

a sequence of one or more additional hidden layers positioned structurally after the sequence of one or more projection layers, the sequence of one or more additional hidden layers configured to receive a layer output generated by a highest projection layer in the sequence of one or more projection layers and to generate one or more additional hidden layer outputs;

wherein execution of the instructions causes the one or more computers to perform operations comprising:
obtaining the projection network input;
inputting the projection network input into the projection neural network; and
receiving the projection network output generated by the projection neural network.

2. The system of claim 1 , wherein the projection neural network further comprises:
an output layer configured to receive the additional hidden layer output generated by a highest additional hidden layer in the sequence of one or more additional hidden layers and to generate the projection network output.

3. The system of claim 1 , wherein the sequence of one or more projection layers comprises a plurality of projection layers positioned structurally one after the other.

4. The system of claim 1 , wherein the sequence of one or more additional hidden layers comprises one or more fully connected layers that each comprise one or more non-linear activations.

5. The system of claim 1 , wherein the sequence of one or more additional hidden layers comprises one or more additional projection layers.

6. The system of claim 1 , wherein the sequence of one or more additional hidden layers comprises one or more projection sequence layers.

7. The system of claim 1 , wherein the projection neural network is trained to perform on-device text classification.

8. The system of claim 7 , wherein execution of the instructions by the one or more computers causes the one or more computers to:
receive an input text;
convert the input text into an intermediate feature vector; and
input the intermediate feature vector as the projection network input to the projection neural network.

9. The system of claim 8 , wherein the intermediate feature vector comprises one or more of the following intermediate features that have been generated from or associated with the input text:
skip-grams;
n-grams;
part of speech tags;
dependency relationships;
knowledge graph information; or
contextual information.

10. The system of claim 1 , wherein, for each projection layer, the plurality of projection layer functions are precomputed and held static.

11. The system of claim 1 , wherein execution of the instructions by the one or more computers causes the one or more computers to:
dynamically compute the plurality of projection layer functions at inference time using one or more seeds.

12. The system of claim 1 , wherein the projection neural network comprises a self-governing neural network that performs natural language processing without initializing, loading, or storing any feature or vocabulary weight matrices.

13. The system of claim 1 , wherein the projection neural network has been trained based solely on its own performance relative to training data.

14. The system of claim 1 , wherein, for each projection layer, each projection function is associated with a respective set of projection vectors, and wherein applying each projection function to the layer input comprises:
for each projection vector:
determining a dot product between the layer input and the projection vector;
when the dot product is negative, assigning a first value to a corresponding position in the projection function output; and
when the dot product is positive, assigning a second value to the corresponding position in the projection function output.

15. The system of claim 1 , wherein, for each projection layer, the projection functions are each encoded as sparse matrices and are used to generate a binary representation from the layer input.

16. The system of claim 1 , wherein the projection layer parameters include a parameter matrix and a bias vector, and wherein generating the layer output by applying the projection layer parameters for the projection layer to the projection function outputs comprises:
applying the parameter matrix to the projection function outputs and then adding the bias vector.

17. A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to implement:
a projection sequence network configured to receive an input and to generate an output from the input, the projection sequence network comprising:
a sequence of one or more projection layers, wherein each projection layer has a plurality of projection layer parameters that are learned during training of the projection sequence network, wherein each projection layer is configured to:
receive a layer input;
apply a plurality of projection layer functions to the layer input, each projection layer function generating a respective projection function output that projects the layer input to a different space, wherein the plurality of projection layer functions comprise a plurality of locality sensitive hashing functions that were held static during training of the projection neural network, and
generate a layer output by applying the projection layer parameters for the projection layer to the projection function outputs; and

one or more projection sequence layers positioned structurally after the sequence of one or more projection layers, wherein each of the one or more projection sequence layers is configured to provide first internal state data to a subsequent iteration of such projection sequence layer in a subsequent iteration of the projection sequence network and to receive second internal state data from the subsequent iteration of such projection sequence layer in the subsequent iteration of the projection sequence network;

wherein execution of the instructions causes the one or more computers to perform operations comprising:
obtaining the input;
inputting the input into the projection sequence network; and
receiving the output generated by the projection sequence network.

18. The system of claim 17 , wherein the input comprises text and the output comprises segments of the text that have been identified and classified into respective ones of a

Assignees

• Google Llc

Inventors

• Ravi Sujith
• KOZAREVA Zornitsa

Classifications

• G06N7/01

  Probabilistic graphical models, e.g. probabilistic networks · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06N3/044

  Recurrent networks, e.g. Hopfield networks · CPC title

• G06N3/0442

  characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] · CPC title

• G06N3/09

  Supervised learning · CPC title