Method and apparatus with neural network processing

Patent metadata

Field	Value
Publication number	US-11823029-B2
Application number	US-202217846202-A
Country	US
Kind code	B2
Filing date	Jun 22, 2022
Priority date	Dec 16, 2019
Publication date	Nov 21, 2023
Grant date	Nov 21, 2023

Abstract

A method of processing a neural network, includes generating an integral map for each channel in a first layer of the neural network based on calculating of area sums of pixel values in first output feature maps of channels in the first layer, generating an accumulated integral map by performing an accumulation operation on the integral maps generated for the respective channels, obtaining pre-output feature maps of a second layer, subsequent to the first layer, by performing a convolution operation between input feature maps of the second layer and weight kernels, and removing offsets in the weight kernels to obtain second output feature maps of the second layer by subtracting accumulated values of the accumulated integral map from pixel values of the pre-output feature maps.

First claim

What is claimed is:

1. A method of processing a neural network in a neural processing apparatus, the method comprising:
generating an integral map for each channel in a first layer of the neural network based on calculating of area sums of pixel values in first output feature maps of channels in the first layer of the neural network;
generating an accumulated integral map by performing an accumulation operation on the integral maps generated for the respective channels;
obtaining pre-output feature maps of a second layer subsequent to the first layer by performing a convolution operation between input feature maps of the second layer and weight kernels; and
obtaining second output feature maps of the second layer by subtracting accumulated values included in the accumulated integral map from pixel values of the pre-output feature maps to remove offsets existing in the weight kernels.

2. The method of claim 1 , wherein the weight kernels comprise weights obtained by asymmetric quantization of the neural network.

3. The method of claim 1 , wherein the integral map comprises data obtained by setting a value obtained by summing pixel values included in an area from a reference pixel of the first output feature map to a first output pixel of the first output feature map to a value of a pixel in an integral map corresponding to the first output pixel.

4. The method of claim 3 , wherein the reference pixel is set to one of four corner pixels of the first output feature map.

5. The method of claim 1 , wherein the accumulated integral map is generated by performing a pixel-wise accumulation operation on the integral maps generated for the respective channels.

6. The method of claim 5 , wherein the accumulated integral map generated from the first layer corresponds to data for offsets of the weight kernels of the second layer generated by asymmetric quantization of the neural network.

7. The method of claim 1 , wherein the obtaining of the second output feature maps comprises:
determining a bounding box of an input feature map mapped to the weight kernel to obtain a pre-output pixel of a pre-output feature map of the second layer;
obtaining pixel values set to pixels of the accumulated integral map corresponding to four corner pixels of the bounding box;
calculating an offset existing in a pixel value of the pre-output pixel based on the obtained pixel values; and
obtaining a second output pixel of a second output feature map by subtracting the calculated offset from the pixel value of the pre-output pixel.

8. The method of claim 7 , wherein
the four corner pixels of the bounding box comprise an upper right pixel, an upper left pixel, a lower right pixel, and a lower left pixel of the bounding box, and
the obtaining of pixel values set to pixels of the accumulated integral map comprises,
obtaining, from the accumulated integral map, a pixel value of a first integral map pixel corresponding to the upper right pixel, a pixel value of a second integral map pixel corresponding to the upper left pixel, a pixel value of a third integral map pixel corresponding to the lower right pixel, and a pixel value of a fourth integral map pixel corresponding to the lower left pixel.

9. The method of claim 8 , wherein, upon the reference pixel for generating the accumulated integral map is the lower left pixel of the first output feature map, the calculating of the offset comprises calculating the offset existing in the pixel value of the pre-output pixel by subtracting a pixel value of the second integral map pixel and a pixel value of the third integral map pixel from a sum of a pixel value of the first integral map pixel and a pixel value of the fourth integral map pixel.

10. A non-transitory computer readable recording medium storing instructions that, when executed by a processor, causes the processor to control performance of the method of claim 1 .

11. A neural processing apparatus for processing a neural network, the neural processing apparatus comprising:
a memory; and
one or more processors configured to:
generate an integral map for each channel in a first layer of the neural network based on calculating of area sums of pixel values in first output feature maps of channels in the first layer of the neural network;
generate an accumulated integral map by performing an accumulation operation on the integral maps generated for the respective channels;
obtain pre-output feature maps of a second layer subsequent to the first layer by performing a convolution operation between input feature maps of the second layer and weight kernels; and
obtain second output feature maps of the second layer by subtracting accumulated values included in the accumulated integral map from pixel values of the pre-output feature maps to remove offsets existing in the weight kernels.

12. The neural processing apparatus of claim 11 , wherein the weight kernels comprise weights obtained by asymmetric quantization of the neural network.

13. The neural processing apparatus of claim 11 , wherein the integral map comprises data obtained by setting a value obtained by summing pixel values included in an area from a reference pixel of the first output feature map to a first output pixel of the first output feature map to a value of a pixel in an integral map corresponding to the first output pixel.

14. The neural processing apparatus of claim 13 , wherein the reference pixel is set to one of four corner pixels of the first output feature map.

15. The neural processing apparatus of claim 11 , wherein the accumulated integral map is generated by performing a pixel-wise accumulation operation on the integral maps generated for the respective channels.

16. The neural processing apparatus of claim 15 , wherein the accumulated integral map generated from the first layer corresponds to data for offsets of the weight kernels of the second layer generated by the asymmetric quantization of the neural network.

17. The neural processing apparatus of claim 11 , wherein the one or more processors are further configured to:
determine a bounding box of an input feature map mapped to the weight kernel to obtain a pre-output pixel of a pre-output feature map of the second layer;
obtain pixel values set to pixels of the accumulated integral map corresponding to four corner pixels of the bounding box;
calculate an offset existing in a pixel value of the pre-output pixel based on the obtained pixel values; and
obtain a second output pixel of a second output feature map by subtracting the calculated offset from the pixel value of the pre-output pixel.

18. The neural processing apparatus of claim 17 , wherein
the four corner pixels of the bounding box comprise an upper right pixel, an upper left pixel, a lower right pixel, and a lower left pixel of the bounding box, and
the one or more processors are further configured to obtain,
from the accumulated integral map, a pixel value of a first integral map pixel corresponding to the upper right pixel, a pixel value of a second integral map pixel corresponding to the upper left pixel, a pixel value of a third integral map pixel corresponding to the lower right pixel, and a pixel value of a fourth integral map pixel corresponding to the lower left pixel.

19. The neural processing apparatus of claim 18 , wherein the one or more processors are further configured to calculate the offset existing in the pixel value of the pre-output pixel by subtracting a pixel value of the second integral map pixel and a pixel value of the third integral map pixe

Assignees

• Samsung Electronics Co Ltd

Inventors

• Ha Sangwon

Classifications

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/0495

  Quantised networks; Sparse networks; Compressed networks · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06N3/08 Primary

  Learning methods · CPC title

• G06N3/048 Primary

  Activation functions · CPC title