Systems and methods of distributed optimization

Patent metadata

Field	Value
Publication number	US-11120102-B2
Application number	US-202017004324-A
Country	US
Kind code	B2
Filing date	Aug 27, 2020
Priority date	Oct 16, 2015
Publication date	Sep 14, 2021
Grant date	Sep 14, 2021

Abstract

Systems and methods of determining a global model are provided. In particular, one or more local updates can be received from a plurality of user devices. Each local update can be determined by the respective user device based at least in part on one or more data examples stored on the user device. The one or more data examples stored on the plurality of user devices are distributed on an uneven basis, such that no user device includes a representative sample of the overall distribution of data examples. The local updates can then be aggregated to determine a global model.

First claim

What is claimed is:

1. A device for performing voice recognition; the device comprising:
a microphone;
a speaker;
one or more processors; and
one or more memory devices, the one or more memory devices storing:
a machine-learned voice recognition model; and
computer-readable instructions that when executed by the one or more processors cause the one or more processors to perform operations, the operations comprising:
determining a local model update associated with an objective function based at least in part on a number of local data examples stored by the device, the local model update being determined by performing a number of iterations of a gradient descent training technique on a local version of the machine-learned voice-recognition model with respect to the number of local data examples stored by the device;
providing the local model update to a central computing system for use in determination of a global version of the machine-learned voice-recognition model based on aggregation of the local model update with one or more additional local model updates received from one or more additional devices, the aggregation of the local model update with the one or more additional local model updates comprising application of a weighting term to each local model update; the weighting term for each local model update being proportional to the number of local data examples stored on the corresponding device;
receiving the global version of the machine-learned voice-recognition model from the central computing system; and
employing the global version of the machine-learned voice-recognition model to produce voice recognition predictions based on voice data captured by the microphone.

2. The device of claim 1 , wherein one or more of the number of local data examples stored by the device are generated through interaction of a user with the device.

3. The device of claim 2 , wherein the one or more of the number of local data examples generated through interaction of the user with the device comprise audio files.

4. The device of claim 1 , wherein the machine-learned voice recognition model comprises a convolutional neural network.

5. A device for performing voice recognition, the device comprising:
a microphone;
a speaker;
one or more processors; and
one or more memory devices, the one or more memory devices storing computer-readable instructions that when executed by the one or more processors cause the one or more processors to perform operations, the operations comprising:
accessing a global version of a machine-learned voice-recognition model, wherein the global version of the machine-learned model has been generated based on aggregation of a plurality of local model updates respectively generated by a plurality of additional devices, the aggregation of the plurality of local model updates comprising application of a weighting term to each local model update, the weighting term for each local model update being proportional to a number of local data examples stored on the additional device that generated the local model update;
capturing voice data with the microphone; and
employing the global version of the machine-learned voice-recognition model to produce voice recognition predictions based on the voice data captured by the microphone.

6. The device of claim 5 , wherein one or more of the number of local data examples stored by each additional device are generated through interaction of a respective user with the additional device.

7. The device of claim 6 , wherein the one or more of the number of local data examples generated through interaction of the respective user with the additional device comprise audio files.

8. The device of claim 5 , wherein the machine-learned voice recognition model comprises a convolutional neural network.

9. A computer-implemented method of performing voice recognition, the method comprising:
receiving, by a device that comprises a microphone and a speaker, a global version of a machine-learned voice-recognition model obtained from a central computing system, wherein the global version of the machine-learned model has been generated based on aggregation of a plurality of local model updates respectively generated by a plurality of additional devices, the aggregation of the plurality of local model updates comprising application of a weighting term to each local model update, the weighting term for each local model update being proportional to a number of local data examples stored on the additional device that generated the local model update; and
capturing, by the device using the microphone, voice data; and
employing, by the device, the received global version of the machine-learned voice recognition model to produce voice recognition predictions based on the voice data captured by the microphone.

10. The computer-implemented method of claim 9 , wherein one or more of the number of local data examples stored by each additional device are generated through interaction of a respective user with the additional device.

11. The computer-implemented method of claim 10 , wherein the one or more of the number of local data examples generated through interaction of the respective user with the additional device comprise audio files.

12. The computer-implemented method of claim 9 , wherein the machine-learned voice recognition model comprises a convolutional neural network.

13. A computer-implemented method of performing voice recognition, the method comprising:
determining, by a first device that comprises a microphone, a speaker, and a machine-learned voice recognition model, a local model update to the machine-learned voice recognition model based at least in part on a gradient of a loss function and a number of locally stored data examples;
providing, by the first device, the local model update to a central computing system for use in determination of a global update to the machine-learned voice recognition model by the central computing system via aggregation of the local model update with one or more additional local model updates received from one or more additional devices, the aggregation of the local model update with the one or more additional local model updates comprising application of a weighting term to each local model update, the weighting term for each local model update being proportional to the number of locally stored data examples stored on the corresponding device;
receiving, by a second device that comprises a microphone and a speaker, the machine-learned voice recognition model from the central computing system after application of the global update to the machine-learned voice recognition model by the central computing system; and
employing, by the second device, the received machine-learned voice recognition model to produce voice recognition predictions based on voice data captured by the microphone of the second device.

14. The computer-implemented method of claim 13 , wherein one or more of the number of locally stored data are generated through interaction of a user with the first device.

15. The computer-implemented method of claim 14 , wherein the one or more of the number of locally stored data are generated through interaction of the user with the first device comprise audio files.

Assignees

• Google Llc

Inventors

• Mcmahan Hugh Brendan
• Konecny Jakub
• Moore Eider Brantly
• Ramage Daniel
• Aguera-Arcas Blaise H

Classifications

• G06N20/00 Primary

  Machine learning · CPC title

• G06F2111/02

  CAD in a network environment, e.g. collaborative CAD or distributed simulation · CPC title

• G06F17/11

  for solving equations {, e.g. nonlinear equations, general mathematical optimization problems (optimization specially adapted for a specific administrative, business or logistic context G06Q10/04)} · CPC title

• G06F17/17 Primary

  Function evaluation by approximation methods, e.g. inter- or extrapolation, smoothing, least mean square method ({G06F17/18 takes precedence } ; interpolation for numerical control G05B19/18) · CPC title

• G06F30/00

  Computer-aided design [CAD] · CPC title