Memory efficient convolution operations in deep learning neural networks

Patent metadata

Field	Value
Publication number	US-11210584-B2
Application number	US-201715420127-A
Country	US
Kind code	B2
Filing date	Jan 31, 2017
Priority date	Jan 31, 2017
Publication date	Dec 28, 2021
Grant date	Dec 28, 2021

Abstract

Input image data having a plurality of pixel values represented in a two-dimensional matrix form of columns and rows is received. The input image data is transformed into a plurality of input rows. The pixel values in each input row correspond to the pixel values in a predetermined subset of the columns of the input image data and all of the rows of each column of the subset of columns. A plurality of subsets of pixel values in the plurality of input rows is determined. The number of pixel values in each row of a subset of pixel values equal in number to a number of filter values in a filter. Each input row of each subset of pixel values is convolved with the filter values of the filter to determine a corresponding output value and stored in a memory.

First claim

What is claimed is:

1. A method comprising:
receiving input image data having a plurality of pixel values represented in a two-dimensional matrix form of columns and rows;
selecting a plurality of subsets of the input image data, each subset comprising a plurality of adjacent columns of the input image data, each subset comprising all rows of the input image data, a number of columns in the plurality of adjacent columns equal to a predetermined width of a two-dimensional filter matrix;
arranging, in a left-to-right and top-to-bottom manner, each of the plurality of subsets into a corresponding linearized input row;
storing the plurality of linearized input rows in a memory;
convolving a first portion of a first linearized input row in the plurality of linearized input rows with filter values of the two-dimensional filter matrix to determine a corresponding first output value, a size of the first portion equal to a size of the two-dimensional filter matrix;
convolving a second portion of the first linearized input row with filter values of the two-dimensional filter matrix to determine a corresponding second output value, a size of the second portion equal to the size of the two-dimensional filter matrix, the second portion determined by shifting within the first linearized input row by a predetermined number of shift values from the first portion; and
storing each output value in the memory in a two-dimensional matrix form.

2. The method of claim 1 , wherein selecting the plurality of subsets comprises:
determining a first subset; and
determining a second subset, the second subset being determined by shifting within each input row by a predetermined number of columns.

3. The method of claim 1 , wherein convolving each linearized input row with the filter values of the two-dimensional filter matrix comprises:
multiplying each pixel value in the linearized input row by a corresponding filter value in the two-dimensional filter matrix; and
summing the results of each multiplication to determine the corresponding output value.

4. The method of claim 3 , further comprising: arranging the filter values of the two-dimensional filter matrix in a column representation prior to the convolving.

5. The method of claim 4 , further comprising transforming the filter values from a two-dimensional matrix representation to the column representation.

6. The method of claim 1 , wherein the convolving is performed in a neural network.

7. A computer usable program product comprising one or more computer-readable storage media, and program instructions stored on at least one of the one or more computer readable storage media, the stored program instructions comprising:
program instructions to receive input image data having a plurality of pixel values represented in a two-dimensional matrix form of columns and rows;
program instructions to select a plurality of subsets of the input image data, each subset comprising a plurality of adjacent columns of the input image data, each subset comprising all rows of the input image data, a number of columns in the plurality of adjacent columns equal to a predetermined width of a two-dimensional filter matrix;
program instructions to arrange, in a left-to-right and top-to-bottom manner, each of the plurality of subsets into a corresponding linearized input row;
program instructions to store the plurality of linearized input rows in a memory;
program instructions to convolve a first portion of a first linearized input row in the plurality of linearized input rows with filter values of the two-dimensional filter matrix to determine a corresponding first output value, a size of the first portion equal to a size of the two-dimensional filter matrix;
program instructions to convolve a second portion of the first linearized input row with filter values of the two-dimensional filter matrix to determine a corresponding second output value, a size of the second portion equal to the size of the two-dimensional filter matrix, the second portion determined by shifting within the first linearized input row by a predetermined number of shift values from the first portion; and
program instructions to store each output value in the memory in a two-dimensional matrix form.

8. The computer usable program product of claim 7 , wherein program instructions to select the plurality of subsets further comprises:
program instructions to determine a first subset; and
program instructions to determine a second subset, the second subset being determined by shifting within each input row by a predetermined number of columns.

9. The computer usable program product of claim 7 , wherein the program instructions to convolve each linearized input row with the filter values of the two-dimensional filter matrix comprise:
program instructions to multiply each pixel value in the linearized input row by a corresponding filter value in the two-dimensional filter matrix; and
program instructions to sum the results of each multiplication to determine the corresponding output value.

10. The computer usable program product of claim 9 , further comprising:
program instructions to arrange the filter values of the two-dimensional filter matrix in a column representation prior to the convolving.

11. The computer usable program product of claim 10 , further comprising program instructions to transform the filter values from a two-dimensional matrix representation to the column representation.

12. A computer system comprising: one or more processors, one or more computer-readable memories, and one or more computer-readable storage media, and program instructions stored on at least one of the one or more computer-readable storage media for execution by at least one of the one or more processors via at least one of the one or more computer-readable memories, the stored program instructions comprising:
program instructions to receive input image data having a plurality of pixel values represented in a two-dimensional matrix form of columns and rows;
program instructions to select a plurality of subsets of the input image data, each subset comprising a plurality of adjacent columns of the input image data, each subset comprising all rows of the input image data, a number of columns in the plurality of adjacent columns equal to a predetermined width of a two-dimensional filter matrix;
program instructions to arrange, in a left-to-right and top-to-bottom manner, each of the plurality of subsets into a corresponding linearized input row;
program instructions to store the plurality of linearized input rows in a memory;
program instructions to convolve a first portion of a first linearized input row in the plurality of linearized input rows with filter values of the two-dimensional filter matrix to determine a corresponding first output value, a size of the first portion equal to a size of the two-dimensional filter matrix;
program instructions to convolve a second portion of the first linearized input row with filter values of the two-dimensional filter matrix to determine a corresponding second output value, a size of the second portion equal to the size of the two-dimensional filter matrix, the second portion determined by shifting within the first linearized input row by a predetermined number of shift values from the first portion; and
program instructions to store each output value in the memory in a two-dimensional matrix form.

13. The computer system of claim 12 , wherein program instructions to select the plurality of subsets further comprises:
program instructions to determine a first subset; and
program instructions to determine a second subset, the

Assignees

• IBM

Inventors

• Brand Daniel
• Cho Minsik

Classifications

• G06N3/045

  Combinations of networks · CPC title

• G06N20/00 Primary

  Machine learning · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06F17/15 Primary

  Correlation function computation {including computation of convolution operations (arithmetic circuits for sum of products per se, e.g. multiply-accumulators G06F7/5443; digital filters, e.g. FIR, IIR, adaptive filters H03H17/00)} · CPC title

• G06N3/08 Primary

  Learning methods · CPC title