Training multiple neural networks with different accuracy

Patent metadata

Field	Value
Publication number	US-2016379113-A1
Application number	US-201615260460-A
Country	US
Kind code	A1
Filing date	Sep 9, 2016
Priority date	May 23, 2014
Publication date	Dec 29, 2016
Grant date	—

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training a deep neural network. One of the methods includes generating a plurality of feature vectors that each model a different portion of an audio waveform, generating a first posterior probability vector for a first feature vector using a first neural network, determining whether one of the scores in the first posterior probability vector satisfies a first threshold value, generating a second posterior probability vector for each subsequent feature vector using a second neural network, wherein the second neural network is trained to identify the same key words and key phrases and includes more inner layer nodes than the first neural network, and determining whether one of the scores in the second posterior probability vector satisfies a second threshold value.

First claim

What is claimed is:

1 . A system comprising:
one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:
training a first neural network to identify a set of features using a first training set, the first neural network comprising a first quantity of nodes;
training a second neural network to identify the set of features using a second training set, the second neural network comprising a second quantity of nodes, greater than the first quantity of nodes; and
providing the first neural network, and the second neural network to a user device that uses both the first neural network and the second neural network to analyze a data set and determine whether the data set comprises a digital representation of a feature from the set of features.

2 . The system of claim 1 , wherein:
the set of features comprises key words and key phrases; and the user device uses both the first neural network and the second neural network to analyze an audio waveform and determine whether a digital representation of one of the key words or the key phrases from the set of features is included in the audio waveform.

3 . The system of claim 2 , the operations comprising:
providing a feature extraction module, a first posterior handling module, and a second posterior handling module to the user device with the first neural network and the second neural network, wherein the user device uses the feature extraction module, the first posterior handling module, and the second posterior handling module to perform the analysis of the data set.

4 . The system of claim 3 , wherein the first posterior handling module and the second posterior handling module comprise the same posterior handling module.

5 . The system of claim 1 , wherein the set of features comprise computer vision features, handwriting recognition features, text classification features, or authentication features.

6 . The system of claim 1 , wherein the first training set and the second training set comprise the same training set.

7 . The system of claim 6 , the operations comprising:
training the first neural network for a first quantity of iterations; and training the second neural network for a second quantity of iterations, greater than the first quantity of iterations.

8 . The system of claim 1 , wherein a ratio between the first quantity of nodes and the second quantity of nodes identifies a performance cost savings of the user device when the user device analyzes a particular portion of the data set with the first neural network and not the second neural network.

9 . A method comprising:
training a first neural network to identify a set of features using a first training set, the first neural network comprising a first quantity of nodes; training a second neural network to identify the set of features using a second training set, the second neural network comprising a second quantity of nodes, greater than the first quantity of nodes; and providing the first neural network, and the second neural network to a user device that uses both the first neural network and the second neural network to analyze a data set and determine whether the data set comprises a digital representation of a feature from the set of features.

10 . The method of claim 9 , wherein:
the set of features comprises key words and key phrases; and the user device uses both the first neural network and the second neural network to analyze an audio waveform and determine whether a digital representation of one of the key words or the key phrases from the set of features is included in the audio waveform.

11 . The method of claim 10 , comprising:
providing a feature extraction module, a first posterior handling module, and a second posterior handling module to the user device with the first neural network and the second neural network, wherein the user device uses the feature extraction module, the first posterior handling module, and the second posterior handling module to perform the analysis of the data set.

12 . The method of claim 11 , wherein the first posterior handling module and the second posterior handling module comprise the same posterior handling module.

13 . The method of claim 9 , wherein the set of features comprise computer vision features, handwriting recognition features, text classification features, or authentication features.

14 . The method of claim 9 , wherein the first training set and the second training set comprise the same training set.

15 . The method of claim 14 , comprising:
training the first neural network for a first quantity of iterations; and training the second neural network for a second quantity of iterations, greater than the first quantity of iterations.

16 . The method of claim 9 , wherein a ratio between the first quantity of nodes and the second quantity of nodes identifies a performance cost savings of the user device when the user device analyzes a particular portion of the data set with the first neural network and not the second neural network.

17 . A non-transitory computer-readable medium storing software comprising instructions executable by one or more computers which, upon such execution, cause the one or more computers to perform operations comprising:
training a first neural network to identify a set of features using a first training set, the first neural network comprising a first quantity of nodes; training a second neural network to identify the set of features using a second training set, the second neural network comprising a second quantity of nodes, greater than the first quantity of nodes; and providing the first neural network, and the second neural network to a user device that uses both the first neural network and the second neural network to analyze a data set and determine whether the data set comprises a digital representation of a feature from the set of features.

18 . The computer-readable medium of claim 17 , wherein:
the set of features comprises key words and key phrases; and the user device uses both the first neural network and the second neural network to analyze an audio waveform and determine whether a digital representation of one of the key words or the key phrases from the set of features is included in the audio waveform.

19 . The computer-readable medium of claim 18 , the operations comprising:
providing a feature extraction module, a first posterior handling module, and a second posterior handling module to the user device with the first neural network and the second neural network, wherein the user device uses the feature extraction module, the first posterior handling module, and the second posterior handling module to perform the analysis of the data set.

20 . The computer-readable medium of claim 19 , wherein the first posterior handling module and the second posterior handling module comprise the same posterior handling module.

Assignees

• Google Inc

Inventors

• Gruenstein Alexander H

Classifications

• G06N3/047

  Probabilistic or stochastic networks · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G10L15/16

  using artificial neural networks · CPC title

• G06F16/367

  Ontology · CPC title

• G06N3/08 Primary

  Learning methods · CPC title