Neural network execution block using fully connected layers

Patent metadata

Field	Value
Publication number	US-11922294-B2
Application number	US-202016854564-A
Country	US
Kind code	B2
Filing date	Apr 21, 2020
Priority date	May 22, 2019
Publication date	Mar 5, 2024
Grant date	Mar 5, 2024

Abstract

Systems and components for use with neural networks. An execution block and a system architecture using that execution block are disclosed. The execution block uses a fully connected stack of layers and one output is a forecast for a time series while another output is a backcast that can be used to determine a residual from the input to the execution block. The execution block uses a waveform generator sub-unit whose parameters can be judiciously selected to thereby constrain the possible set of waveforms generated. By doing so, the execution block specializes its function. The system using the execution block has been shown to be better than the state of the art in providing solutions to the time series problem.

First claim

We claim:

1. A non-transitory computer readable medium having code stored thereon to provide an execution block for use with an artificial intelligence neural network system for time series forecasting, the execution block comprising:
a stack of fully connected layers of neural network nodes, said stack having input and output, said output being received in parallel by a first parallel branch and a second parallel branch;
said first parallel branch comprising:
a first fully connected layer of neural network nodes receiving and processing said output and a first waveform generator sub-unit receiving an output of said first fully connected layer;

said second parallel branch comprising:
a second fully connected layer of neural network nodes receiving and processing said output and a second waveform generator sub-unit receiving an output of said second fully connected layer;

wherein
an output of said first parallel branch is a forecast of said execution block based on said input computed as a synthesis of basis functions of said execution block;
an output of said second parallel branch is an estimate of said execution block for said input and is used to form a residual of said input to said execution block;
different basis stacks are trained using different training data sets such that different basis stacks are suitable for different tasks; and
said execution block is deployable for use with input windows of different lengths absent specific modification of the execution block for said input windows.

2. The non-transitory computer readable medium according to claim 1 , wherein each of said first waveform generator sub-unit and second waveform generator sub-unit implements a function that maps a set of points in a time domain to a set of points in a forecast value domain.

3. The non-transitory computer readable medium according to claim 1 , wherein said execution block is used to forecast a time series output.

4. The non-transitory computer readable medium according to claim 3 , wherein an input to said execution block is an input signal detailing a history lookback window of values of a time series.

5. The non-transitory computer readable medium according to claim 2 , wherein an output of each of said first waveform generator sub-unit and second waveform generator sub-unit is based on a set of parameters.

6. The non-transitory computer readable medium according to claim 3 , wherein an output of each of said first waveform generator sub-unit and second waveform generator sub-unit encodes an inductive bias to regularize and constrain a structure of viable solutions for a time series problem.

7. The non-transitory computer readable medium according to claim 3 , wherein an output of each of said first waveform generator sub-unit and second waveform generator sub-unit is based on a plurality of time varying waveforms.

8. The non-transitory computer readable medium according to claim 7 wherein said waveforms are selected based on a set of parameters selected for said first waveform generator sub-unit or second waveform generator sub-unit.

9. The non-transitory computer readable medium according to claim 2 , wherein said execution block is used in a neural network system for time series forecasting.

10. The non-transitory computer readable medium according to claim 1 wherein the neural network system is used for providing a forecast related to one or more of the group of industries consisting of: economic, finance, demographics and industry, tourism, trade, labor and wage, real estate, transportation, and natural resources and environment.

11. A non-transitory computer readable medium having code stored thereon to provide an artificial intelligence neural network system for use in time series forecasting, the system comprising:
a plurality of basis stacks, said basis stacks being coupled in sequence with each basis stack comprising at least two execution blocks, an output of each basis stack being added to a cumulative output for said neural network system;
wherein
each of said at least two execution blocks comprises:
a stack of fully connected layers of neural network nodes, said stack having input and output, said output being received in parallel by a first parallel branch and a second parallel branch;
said first parallel branch comprising:
a first fully connected layer of neural network nodes receiving and processing said output and a first waveform generator sub-unit receiving an output of said first fully connected layer;

said second parallel branch comprising:
a second fully connected layer of neural network nodes receiving and processing said output and a second waveform generator sub-unit receiving an output of said second fully connected layer;

and wherein
an output of said first parallel branch is a forecast of said execution block based on said input computed as a synthesis of basis functions of said execution block;
an output of said second parallel branch is an estimate of said execution block for said input and is used to form a residual of said execution block; and
each of said at least two execution blocks is forced into a specialization by constraining a range of wave functions generated by the corresponding first and second waveform generator sub-units;
different basis stacks are trained using different training data sets such that different basis stacks are suitable for different tasks; and
said artificial intelligence neural network system is deployable for use with input windows of different lengths absent specific modification of the execution block for said input windows.

12. The non-transitory computer readable medium according to claim 11 , wherein each of said first waveform generator sub-unit and second waveform generator sub-unit implements a function that maps a set of points in a time domain to a set of points in a forecast value domain.

13. The non-transitory computer readable medium according to claim 12 , wherein an input to said neural network system is an input signal detailing a history lookback window of values of a time series.

14. The non-transitory computer readable medium according to claim 12 , wherein an output of each of said first waveform generator sub-unit and second waveform generator sub-unit is based on a set of parameters.

15. The non-transitory computer readable medium according to claim 12 , wherein an output of each of said first waveform generator sub-unit and second waveform generator sub-unit encodes an inductive bias to regularize and constrain a structure of viable solutions for a time series problem.

16. The non-transitory computer readable medium according to claim 12 , wherein an output of each of said first waveform generator sub-unit and second waveform generator sub-unit is based on a plurality of time varying waveforms.

17. The non-transitory computer readable medium according to claim 16 , wherein said waveforms are selected based on a set of parameters selected for said first waveform generator sub-unit or second waveform generator sub-unit.

18. The non-transitory computer readable medium according to claim 11 wherein the neural network system is used for providing a forecast related to one or more of the group of industries consisting of: economic, finance, demographics and industry, tourism, trade, labor and wage, real estate, transportation, and natural resources and environment.

Assignees

• Element Ai Inc
• Servicenow Canada Inc

Inventors

• Oreshkin Boris
• Carpov Dmitri

Classifications

• G06N3/09

  Supervised learning · CPC title

• G06N3/0499

  Feedforward networks · CPC title

• G06N3/063 Primary

  using electronic means · CPC title

• G06F1/0321

  Waveform generators, i.e. devices for generating periodical functions of time, e.g. direct digital synthesizers (G06F1/0314, G06F1/035 take precedence) · CPC title

• G06F17/18

  for evaluating statistical data {, e.g. average values, frequency distributions, probability functions, regression analysis (forecasting specially adapted for a specific administrative, business or logistic context G06Q10/04)} · CPC title