Memory bandwidth management for deep learning applications

Patent metadata

Field	Value
Publication number	US-2016379111-A1
Application number	US-201514750277-A
Country	US
Kind code	A1
Filing date	Jun 25, 2015
Priority date	Jun 25, 2015
Publication date	Dec 29, 2016
Grant date	—

Abstract

In a data center, neural network evaluations can be included for services involving image or speech recognition by using a field programmable gate array (FPGA) or other parallel processor. The memory bandwidth limitations of providing weighted data sets from an external memory to the FPGA (or other parallel processor) can be managed by queuing up input data from the plurality of cores executing the services at the FPGA (or other parallel processor) in batches of at least two feature vectors. The at least two feature vectors can be at least two observation vectors from a same data stream or from different data streams. The FPGA (or other parallel processor) can then act on the batch of data for each loading of the weighted datasets.

First claim

What is claimed is:

1 . A method of performing neural network processes, the method comprising:
receiving, at a field programmable gate array (FPGA), a batch of input data for accelerated processing of a neural network evaluation, wherein the batch of input data comprises at least two feature vectors; loading the FPGA with a first layer set of weights for the neural network evaluation from an external memory; and applying, within the FPGA, the first layer set of weights to the batch of the input data to generate intermediates.

2 . The method of claim 1 , wherein the at least two feature vectors comprise one observation vector from each of at least two data streams.

3 . The method of claim 2 , wherein the neural network evaluation is a recurrent neural network evaluation.

4 . The method of claim 1 , wherein the at least two feature vectors comprise at least two observation vectors from each of at least two data streams.

5 . The method of claim 1 , wherein the at least two feature vectors comprise at least two observation vectors from a single data stream.

6 . The method of claim 1 , further comprising:
after applying the first layer set of weights to the batch, loading the FPGA with a second layer set of weights for the neural network evaluation from the external memory; and applying, within the FPGA, the second layer set of weights to the intermediates.

7 . The method of claim 1 , wherein the neural network evaluation is a deep neural network multi-layer perceptron evaluation.

8 . One or more computer readable storage media having instructions stored thereon that when executed by a processing system, direct the processing system to manage memory bandwidth for deep learning applications by:
directing a batch of at least two observation vectors from at least one core to queue up at a field programmable gate array (FPGA); loading at least one weighted dataset on the FPGA, each of the at least one weighted dataset being loaded once per batch of the at least two observation vectors directed to queue up at the FPGA; and directing an evaluation output from the FPGA to the at least one core for further processing.

9 . The media of claim 8 , wherein the instructions that direct the batch of the at least two observation vectors from at least one core to queue up at the FPGA direct one observation vector from each of at least two cores to queue up at the FPGA.

10 . The media of claim 8 , wherein the instructions that direct the batch of the at least two observation vectors from at least one core to queue up at the FPGA direct at least two observation vectors from each of at least two cores to queue up at the FPGA.

11 . A system comprising:
one or more storage media; a plurality of processing cores; a service, the service being stored on at least one of the one or more storage media and executed on at least the plurality of processing cores; a parallel processor in communication with the plurality of cores to perform a neural network evaluation on a batch of data for a process of the service; and weight datasets for the neural network evaluation stored on at least one of the one or more storage media.

12 . The system of claim 11 , wherein the parallel processor is a field programmable gate array (FPGA).

13 . The system of claim 11 , wherein the parallel processor receives one observation vector from each core of the plurality of cores as the batch of data.

14 . The system of claim 13 , wherein the neural network evaluation comprises a recurrent neural network evaluation.

15 . The system of claim 11 , wherein the neural network evaluation comprises a deep neural network multi-layer perceptron evaluation.

16 . The system of claim 11 , wherein the parallel processor receives at least two observation vectors from each core of the plurality of cores as the batch of data.

17 . The system of claim 11 , wherein the service comprises a speech recognition service.

18 . The system of claim 11 , further comprising:
a manager agent stored, at least in part, on at least one of the one or more storage media, that when executed, directs the system to: direct the batch of data from at least one of the plurality of processing cores to queue up at the parallel processor; load at least one weighted dataset of the weight datasets onto the parallel processor, each of the at least one weighted dataset being loaded once per batch; and direct an evaluation output from the parallel processor to the plurality of processing cores.

19 . The system of claim 18 , wherein the manager agent directs the system to direct the batch of data to queue up at the parallel processor by directing at least one observation vector from of each of at least two cores of the plurality of cores to the parallel processor.

20 . The system of claim 18 , wherein the manager agent directs the system to direct the batch of data to queue up at the parallel processor by directing at least two observation vectors from of each of at least two cores of the plurality of cores to the parallel processor.

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Bittner Jr Ray A
• Seide Frank Torsten Bernd

Classifications

• G06N3/044 Primary

  Recurrent networks, e.g. Hopfield networks · CPC title

• G06N3/063 Primary

  using electronic means · CPC title

• G06N3/04

  Architecture, e.g. interconnection topology · CPC title

• G06N3/08 Primary

  Learning methods · CPC title

• G06N3/0499

  Feedforward networks · CPC title