Application development platform and software development kits that provide comprehensive machine learning services

Patent metadata

Field	Value
Publication number	US-11410044-B2
Application number	US-201816605702-A
Country	US
Kind code	B2
Filing date	May 21, 2018
Priority date	May 20, 2017
Publication date	Aug 9, 2022
Grant date	Aug 9, 2022

Abstract

The present disclosure provides an application development platform and associated software development kits (“SDKs”) that provide comprehensive services for generation, deployment, and management of machine-learned models used by computer applications such as, for example, mobile applications executed by a mobile computing device. In particular, the application development platform and SDKs can provide or otherwise leverage a unified, cross-platform application programming interface (“API”) that enables access to all of the different machine learning services needed for full machine learning functionality within the application. In such fashion, developers can have access to a single SDK for all machine learning services.

First claim

What is claimed is:

1. A computing system for jointly learning compact models, the computing system comprising:
one or more processors; and
one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising:
obtaining a pre-trained machine-learned model associated with a user;
obtaining one or more user selections associated with the user;
generating a training pipeline comprising one or more user-created schema based on the one or more user selections;
generating a compact model, wherein the compact model comprises an inference speed parameter defined by the user; and
jointly training the compact model and the pre-trained machine-learned model on a set of training data, wherein the compact model has a smaller model size relative to the pre-trained machine-learned model, wherein jointly training the compact model and the pre-trained machine-learned model comprises backpropagating a gradient of a loss function, wherein the gradient is based on a first comparison between an output of the compact model for an input and a ground truth output and a second comparison between the output of the compact model and an output of the pre-trained machine-learned model, wherein jointly training reduces a number of bits used for one or more model weights and one or more activations, and wherein jointly training comprises:
validating the one or more user-created schema of a training pipeline; and
training the compact model and the pre-trained machine-learned model based at least in part on the user-created schema.

2. The computing system of claim 1 , wherein jointly training the compact model and the pre-trained machine-learned model comprises backpropagating the gradient of the loss function, wherein the loss function comprises a combined loss function that comprises:
a first loss term that describes a trainer prediction error;
a second loss term that describes a student simulation error; and
a third loss term that describes a student prediction error.

3. The computing system of claim 2 , wherein the first loss term describes a distance between the output of the pre-trained machine-learned model and a ground truth.

4. The computing system of claim 2 , wherein the second loss term describes a distance between the output of the pre-trained machine-learned model and the output of the compact model.

5. The computing system of claim 2 , wherein the combined loss function comprises first, second, and third weighting values that respectively weight the first, second, and third loss terms relative to each other.

6. The computing system of claim 1 , wherein the operations further comprise:
receiving a user input that selects one of a number of available compact model architectures;
wherein generating the compact model comprises generating the compact model that has the compact model architecture selected by the user input.

7. The computing system of claim 1 , wherein the operations further comprise:
receiving a user input that specifies one or both of a desired model size or a desired inference speed;
wherein generating the compact model comprises generating the compact model that exhibits one or more both of the desired model size or the desired inference speed.

8. The computing system of claim 1 , wherein the operations further comprise:
receiving a user input that specifies a type of input data;
wherein generating the compact model comprises generating the compact model that is configured to receive and process the type of input data specified by the user input.

9. The computing system of claim 1 , wherein the operations further comprise:
receiving a user input that specifies a prediction task;
wherein generating the compact model comprises generating the compact model that is configured to perform the prediction task.

10. The computing system of claim 1 , wherein:
generating the compact model comprises generating a plurality of compact models that have different architectures; and
jointly training the compact model and the pre-trained machine-learned model comprises jointly training the plurality of compact models and the pre-trained machine-learned model.

11. The computing system of claim 10 , wherein the operations further comprise, after jointly training the plurality of compact models, deploying the plurality of compact models respectively to a plurality of different user groups that have different user device capabilities.

12. The computing system of claim 1 , wherein the compact model comprises a projection neural network.

13. The computing system of claim 1 , wherein the pre-trained machine-learned model comprises a checkpoint from a production model deployed for cloud-based inference.

14. The computing system of claim 1 , wherein the pre-trained machine-learned model is selected by the user from a set of provided pre-trained models.

15. The computing system of claim 1 , wherein the operations further comprise:
determining one or more particular weights of the compact model that are low-scoring weights; and
removing the one or more particular weights of the compact model.

16. A computer-implemented method, comprising:
obtaining, by one or more computing devices, a pre-trained machine-learned model associated with a user;
receiving, by the one or more computing devices, user input that describes one or more desired model characteristics, the one or more desired model characteristics comprising one or more of: a desired model size, a desired inference speed, a type of input data, or a prediction type;
generating, by the one or more computing devices, a training pipeline comprising one or more user-created schema based on the user input, wherein the user input comprises one or more selections associated with the user;
automatically generating, by the one or more computing devices, a compact model specification based at least in part on the user input, wherein the compact model comprises an inference speed parameter defined by the user; and jointly training, by the one or more computing devices, a compact model that has the compact model specification and the pre-trained machine-learned model on a set of training data, wherein the compact model has a smaller model size relative to the pre-trained machine-learned model, and wherein the compact model exhibits the one or more desired model characteristics, wherein jointly training the compact model and the pre-trained machine-learned model comprises backpropagating a gradient of a loss function, wherein the gradient is based on a first comparison between an output of the compact model for an input and a ground truth output and a second comparison between the output of the compact model and an output of the pre-trained machine-learned model, wherein jointly training reduces a number of bits used for one or more model weights and one or more activations, and wherein jointly training comprises:
validating the one or more user-created schema of a training pipeline; and
training the compact model and the pre-trained machine-learned model based at least in part on the user-created schema.

17. The computer-implemented method of claim 16 , wherein jointly training, by the one or more computing devices, the compact model that has the compact model specification and the pre-trained machine-learned model comprises backpropagating, by the one or more computing devices, a gradient of a combined loss function that comprises:
a first loss term that describes a trainer predictio

Assignees

• Google Llc

Inventors

• Ravi Sujith
• Menghani Gaurav
• Kaliamoorthi Prabhu
• Fan Yicheng

Classifications

• G06N3/045

  Combinations of networks · CPC title

• G06F18/214

  Generating training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06N3/096

  Transfer learning · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/0495

  Quantised networks; Sparse networks; Compressed networks · CPC title