Statistical-analysis-based reset of recurrent neural networks for automatic speech recognition

Patent metadata

Field	Value
Publication number	US-10255909-B2
Application number	US-201715637559-A
Country	US
Kind code	B2
Filing date	Jun 29, 2017
Priority date	Jun 29, 2017
Publication date	Apr 9, 2019
Grant date	Apr 9, 2019

Abstract

Techniques are provided for calculating reset parameters for recurrent neural networks (RNN). A methodology implementing the techniques according to an embodiment includes generating a sequence of statistics. The calculation of each statistic is based on outputs of an RNN that is periodically re-initialized at a selected RNN reset time such that each of the calculated statistics is associated with a unique RNN reset time selected from a pre-determined range of reset times. The method further includes analyzing the sequence to identify a maximum interval during which the sequence remains relatively constant. The method further includes selecting a reset time parameter and reset context duration parameter, for re-initialization of the RNN during operation. The reset time parameter is based on the duration of the identified maximum interval and the reset context duration parameter is based on a time associated with the starting point of the identified maximum interval.

First claim

What is claimed is:

1. A processor-implemented method for calculating recurrent neural network (RNN) reset parameters, the method comprising:
calculating, by a processor-based system, a statistic based on outputs of an RNN that is periodically re-initialized at a selected RNN reset time, the calculated statistic associated with the selected RNN reset time;
generating, by the processor-based system, a statistical sequence from a plurality of the calculated statistics wherein each of the plurality of calculated statistics is associated with a unique selected RNN reset time selected from a pre-determined range of RNN reset times;
analyzing, by the processor-based system, the sequence to identify a maximum interval during which variations of the sequence do not exceed a threshold value;
selecting, by the processor-based system, a reset time parameter for re-initialization of the RNN during operation, the reset time parameter based on the identified maximum interval;
selecting, by the processor-based system, a reset context duration parameter for the re-initialization of the RNN during operation, the reset context duration parameter based on a time associated with a starting point of the identified maximum interval; and
storing, by the processor-based system, the reset time parameter and the reset context duration parameter for use by a reset circuit to re-initialize the RNN during operation;
wherein the RNN is an acoustic model for an automatic speech recognition system.

2. The method of claim 1 , further comprising:
calculating a standard deviation of each of the outputs of the RNN during execution of the RNN after the selected reset time; and
averaging the standard deviations associated with each of the outputs to generate the statistic associated with the selected RNN reset time.

3. The method of claim 1 , further comprising ordering the sequence such that the statistics correspond to RNN reset times of increasing durations.

4. The method of claim 1 , wherein the reset time parameter and reset context duration parameter are optimized relative to general pre-defined values.

5. The method of claim 1 , wherein the RNN implements a long short-term memory (LSTM) architecture.

6. The method of claim 1 , wherein the re-initialization of the RNN employs a quantity of training feature vectors based on the reset context duration parameter and is performed at a periodic interval based on the reset time parameter.

7. The method of claim 1 , wherein the re-initialization of the RNN is performed to maintain stability of the RNN.

8. The method of claim 1 , wherein the reset time parameter and the reset context duration parameter are calculated subsequent to training of the RNN and prior to operation of the RNN.

9. A system for calculating recurrent neural network (RNN) reset parameters, the system comprising:
a statistics calculation circuit to calculate a statistic based on outputs of an RNN that is periodically re-initialized at a selected RNN reset time, the calculated statistic associated with the selected RNN reset time, and to generate a statistical sequence from a plurality of the calculated statistics wherein each of the plurality of calculated statistics is associated with a unique selected RNN reset time selected from a pre-determined range of RNN reset times;
a statistics analysis circuit to analyze the sequence to identify a maximum interval during which variations of the sequence do not exceed a threshold value; and
a reset-parameter calculation circuit to select a reset time parameter for re-initialization of the RNN during operation, the reset time parameter based on the identified maximum interval, select a reset context duration parameter for the re-initialization of the RNN during operation, the reset context duration parameter based on a time associated with a starting point of the identified maximum interval, and store the reset time parameter and the reset context duration parameter for use by a reset circuit to re-initialize the RNN during operation;
wherein the RNN is an acoustic model for an automatic speech recognition system.

10. The system of claim 9 , wherein the statistics calculation circuit is further to calculate a standard deviation of each of the outputs of the RNN during execution of the RNN after the selected reset time, and average the standard deviations associated with each of the outputs to generate the statistic associated with the selected RNN reset time.

11. The system of claim 9 , wherein the statistics calculation circuit is further to order the sequence such that the statistics correspond to RNN reset times of increasing durations.

12. The system of claim 9 , wherein the reset time parameter and reset context duration parameter are optimized relative to general pre-defined values.

13. The system of claim 9 , wherein the RNN implements a long short-term memory (LSTM) architecture.

14. The system of claim 9 , wherein the re-initialization of the RNN employs a quantity of training feature vectors based on the reset context duration parameter and is performed at a periodic interval based on the reset time parameter.

15. The system of claim 9 , wherein the re-initialization of the RNN is performed to maintain stability of the RNN.

16. The system of claim 9 , wherein the reset-parameter calculation circuit is further to calculate the reset time parameter and the reset context duration parameter subsequent to training of the RNN and prior to operation of the RNN.

17. At least one non-transitory computer readable storage medium having instructions encoded thereon that, when executed by one or more processors, result in the following operations for calculating recurrent neural network (RNN) reset parameters, the operations comprising:
calculating a statistic based on outputs of an RNN that is periodically re-initialized at a selected RNN reset time, the calculated statistic associated with the selected RNN reset time;
generating a statistical sequence from a plurality of the calculated statistics wherein each of the plurality of calculated statistics is associated with a unique selected RNN reset time selected from a pre-determined range of RNN reset times;
analyzing the sequence to identify a maximum interval during which variations of the sequence do not exceed a threshold value;
selecting a reset time parameter for re-initialization of the RNN during operation, the reset time parameter based on the identified maximum interval;
selecting a reset context duration parameter for the re-initialization of the RNN during operation, the reset context duration parameter based on a time associated with a starting point of the identified maximum interval; and
storing the reset time parameter and the reset context duration parameter for use by a reset circuit to re-initialize the RNN during operation;
wherein the RNN is an acoustic model for an automatic speech recognition system.

18. The computer readable storage medium of claim 17 , further comprising the operations of:
calculating a standard deviation of each of the outputs of the RNN during execution of the RNN after the selected reset time; and
averaging the standard deviations associated with each of the outputs to generate the statistic associated with the selected RNN reset time.

19. The computer readable storage medium of claim 17 , further comprising the operation of ordering the sequence such that the statistics correspond to RNN reset times of increasing durations.

20. The computer readable storage medium of claim 17 , wher

Assignees

• Intel Ip Corp

Inventors

• Hofer Joachim
• Bauer Josef G
• Rozen Piotr
• Stemmer Georg

Classifications

• G10L15/063

  Training · CPC title

• G06N3/044

  Recurrent networks, e.g. Hopfield networks · CPC title

• G06N3/08 Primary

  Learning methods · CPC title

• G10L15/16 Primary

  using artificial neural networks · CPC title

• G10L25/30

  using neural networks · CPC title