Application Development Platform and Software Development Kits that Provide Comprehensive Machine Learning Services

Patent metadata

Field	Value
Publication number	US-2020125956-A1
Application number	US-201816605702-A
Country	US
Kind code	A1
Filing date	May 21, 2018
Priority date	May 20, 2017
Publication date	Apr 23, 2020
Grant date	—

Abstract

The present disclosure provides an application development platform and associated software development kits (“SD-Ks”) that provide comprehensive services for generation, deployment, and management of machine-learned models used by computer applications such as, for example, mobile applications executed by a mobile computing device. In particular, the application development platform and SDKs can provide or otherwise leverage a unified, cross-platform application programming interface (“API”) that enables access to all of the different machine learning services needed for full machine learning functionality within the application. In such fashion, developers can have access to a single SDK for all machine learning services.

First claim

1 . A computing system for jointly learning compact models, the computing system comprising:
one or more processors; and one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising:
obtaining a pre-trained machine-learned model associated with a user;
generating a compact model; and
jointly training the compact model and the pre-trained machine-learned model on a set of training data, wherein the compact model has a smaller model size relative to the pre-trained machine-learned model.

2 . The computing system of claim 1 , wherein jointly training the compact model and the pre-trained machine-learned model comprises backpropagating a gradient of a combined loss function that comprises:
a first loss term that describes a trainer prediction error; a second loss term that describes a student simulation error; and a third loss term that describes a student prediction error.

3 . The computing system of claim 2 , wherein the first loss term describes a distance between an output of the pre-trained machine-learned model and a ground truth.

4 . The computing system of claim 2 , wherein the second loss term describes a distance between an output of the pre-trained machine-learned model and an output of the compact model.

5 . The computing system of claim 2 , wherein the third loss term describes a distance between an output of the compact model and a ground truth.

6 . The computing system of claim 2 , wherein the combined loss function comprises first, second, and third weighting values that respectively weight the first, second, and third loss terms relative to each other.

7 . The computing system of claim 1 , wherein the operations further comprise:
receiving a user input that selects one of a number of available compact model architectures; wherein generating the compact model comprises generating the compact model that has the compact model architecture selected by the user input.

8 . The computing system of claim 1 , wherein the operations further comprise:
receiving a user input that specifies one or both of a desired model size or a desired inference speed; wherein generating the compact model comprises generating the compact model that exhibits one or more both of the desired model size or the desired inference speed.

9 . The computing system of claim 1 , wherein the operations further comprise:
receiving a user input that specifies a type of input data; wherein generating the compact model comprises generating the compact model that is configured to receive and process the type of input data specified by the user input.

10 . The computing system of claim 1 , wherein the operations further comprise:
receiving a user input that specifies a prediction task; wherein generating the compact model comprises generating the compact model that is configured to perform the prediction task.

11 . The computing system of claim 1 , wherein:
generating the compact model comprises generating a plurality of compact models that have different architectures; and jointly training the compact model and the pre-trained machine-learned model comprises jointly training the plurality of compact models and the pre-trained machine-learned model.

12 . The computing system of claim 11 , wherein the operations further comprise, after jointly training the plurality of compact models, deploying the plurality of compact models respectively to a plurality of different user groups that have different user device capabilities

13 . The computing system of claim 1 , wherein the compact model comprises a projection neural network.

14 . The computing system of claim 1 , wherein the compact model comprises a bi-directional LSTM model, a depth-separable ConvNet model, a MobileNet model, a NASNet model, an Inception model, or a ResNet model.

15 . The computing system of claim 1 , wherein the training data comprises user-supplied training data.

16 . The computing system of claim 1 , wherein the pre-trained machine-learned model comprises a checkpoint from a production model deployed for cloud-based inference.

17 . The computing system of claim 1 , wherein the pre-trained machine-learned model is selected by the user from a set of provided pre-trained models.

18 . A computer-implemented method, comprising:
obtaining, by one or more computing devices, a pre-trained machine-learned model associated with a user; receiving, by the one or more computing devices, user input that describes one or more desired model characteristics, the one or more desired model characteristics comprising one or more of: a desired model size, a desired inference speed, a type of input data, or a prediction type; automatically generating, by the one or more computing devices, a compact model specification based at least in part on the user input; and jointly training, by the one or more computing devices, a compact model that has the compact model specification and the pre-trained machine-learned model on a set of training data, wherein the compact model has a smaller model size relative to the pre-trained machine-learned model, and wherein the compact model exhibits the one or more desired model characteristics.

19 . The computer-implemented method of claim 18 , wherein jointly training, by the one or more computing devices, the compact model that has the compact model specification and the pre-trained machine-learned model comprises backpropagating, by the one or more computing devices, a gradient of a combined loss function that comprises:
a first loss term that describes a trainer prediction error exhibited between an output the pre-trained machine-learned model and a ground truth; a second loss term that describes a student simulation error exhibited between the output of the pre-trained machine-learned model and an output of the compact model; and a third loss term that describes a student prediction error exhibited between the output of the compact model and the ground truth.

20 . One or more non-transitory computer-readable media that collectively store instructions that, when executed by one or more processors, cause the one or more processors to perform operations, the operations comprising:
obtaining a pre-trained machine-learned model associated with a user; receiving user input that describes one or more desired model characteristics, the one or more desired model characteristics comprising one or more of: a desired model size, a desired inference speed, a type of input data, or a prediction type; automatically generating a compact model specification based at least in part on the user input; and jointly training a compact model that has the compact model specification and the pre-trained machine-learned model on a set of training data, wherein the compact model has a smaller model size relative to the pre-trained machine-learned model, and wherein the compact model exhibits the one or more desired model characteristics.

Assignees

• Google Llc

Inventors

• Ravi Sujith
• Menghani Gaurav
• Kaliamoorthi Prabhu
• Fan Yicheng

Classifications

• G06N3/084 Primary

  Backpropagation, e.g. using gradient descent · CPC title

• G06F8/76

  Adapting program code to run in a different environment; Porting · CPC title

• G06F9/541

  via adapters, e.g. between incompatible applications · CPC title

• G06K9/6256

  Physics · mapped topic

• G06F18/214

  Generating training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title