Method and device with convolution neural network processing

Patent metadata

Field	Value
Publication number	US-11836971-B2
Application number	US-201916549002-A
Country	US
Kind code	B2
Filing date	Aug 23, 2019
Priority date	Aug 23, 2018
Publication date	Dec 5, 2023
Grant date	Dec 5, 2023

Abstract

A processor-implemented method implementing a convolution neural network includes: determining a plurality of differential groups by grouping a plurality of raw windows of an input feature map into the plurality of differential groups; determining differential windows by performing, for each respective differential group of the differential groups, a differential operation between the raw windows of the respective differential group; determining a reference element of an output feature map corresponding to a reference raw window among the raw windows by performing a convolution operation between a kernel and the reference raw window; and determining remaining elements of the output feature map by performing a reference element summation operation based on the reference element and each of a plurality of convolution operation results determined by performing respective convolution operations between the kernel and each of the differential windows.

First claim

What is claimed is:

1. A processor-implemented method implementing a convolution neural network, the method comprising:
generating a plurality of differential groups by grouping a plurality of raw windows of an input feature map into the plurality of differential groups, the plurality of raw windows are sub-feature maps of the input feature map;
generating differential windows by performing, for each respective differential group of the differential groups, a differential operation between raw windows of the respective differential group, the differential operation being an element-wise differential operation between two raw windows of each respective differential group;
determining a reference element of an output feature map corresponding to a reference raw window among the raw windows by performing a convolution operation between a kernel and the reference raw window; and
determining remaining elements of the output feature map by performing a reference element summation operation based on the reference element and each of a plurality of convolution operation results determined by performing respective convolution operations between the kernel and each of the differential windows.

2. The method of claim 1 ,
wherein the raw windows are determined from the input feature map according to a sliding window fashion,
wherein the grouping of the raw windows into the differential groups comprises grouping, for each respective differential group, two of the raw windows into the respective differential group, and
wherein the two of the raw windows are adjacent in a stride direction of to the sliding window fashion.

3. The method of claim 2 , wherein the element-wise differential operation is being performed between the two adjacent raw windows of the respective differential group.

4. The method of claim 2 , wherein the performing of the summation operation comprises performing the summation operation between the reference element and each of a plurality of cascading summation results determined by performing a cascading summation operation on each of the convolution operation results.

5. The method of claim 4 , wherein
the performing of the cascading summation operation on one of the convolution operation results comprises summing the one of the convolution operation results and other ones of the convolution operation results,
wherein the other ones of the convolution operation results correspond to one or more differential windows preceding, in the sliding direction, a differential window corresponding to the one of the convolution operation results.

6. The method of claim 1 , further comprising:
in response to each of the differential windows being a bit data format, converting each of the differential windows into a data format comprising information about a significant bit digit representing a bit value of 1,
wherein the convolution operation results between the kernel and each of the differential windows are convolution operation results between the kernel and each of the differential windows converted into the data format.

7. The method of claim 6 , wherein the convolution operation results between the kernel and each of the differential windows converted into the data format are results calculated according to a bit-shift operation result based on the information about the significant bit digit.

8. The method of claim 7 , further comprising preprocessing the differential windows according to a booth algorithm that reduces significant bit digits, as pre-processing for converting the data format.

9. The method of claim 1 , further comprising:
performing, using the convolution neural network, any one or any combination of any two or more of denoising, demosaicking, sharpening, deblurring, and super-resolution imaging of an image based on the output feature map,
wherein the convolution neural network is a computational imaging deep neural network (CI-DNN).

10. A non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform the method of claim 1 .

11. The method of claim 1 , wherein each respective differential group includes a first raw window and a second raw window of the input feature map,
wherein the differential operation performed based on an element-wise difference between the first raw window and the second raw window of each respective differential group.

12. A neural network processing device comprising:
one or more neural network processors configured to:
generate a plurality of differential groups by grouping a plurality of raw windows of an input feature map into the plurality of differential groups, the plurality of raw windows are sub-feature maps of the input feature map;
generate differential windows by performing, for each respective differential group, a differential operation between raw windows of the respective differential group, the differential operation being an element-wise differential operation between two raw windows of each respective differential group;
determine a reference element of an output feature map corresponding to a reference raw window among the raw windows by performing a convolution operation between a kernel and the reference raw window; and
determine remaining elements of the output feature map by performing a reference element summation operation based on the reference element and each of a plurality of convolution operation results determined by performing a convolution operation between the kernel and each of the differential windows.

13. The device of claim 12 ,
wherein the raw windows are determined from the input feature map according to a sliding window fashion,
wherein the grouping of the raw windows into the differential groups comprises grouping, for each respective differential group, two of the raw windows into the respective differential group, and
wherein the two of the raw windows are adjacent in a stride direction of to the sliding window fashion.

14. The device of claim 13 , wherein, the element-wise differential operation is being performed between the two adjacent raw windows of the respective differential group.

15. The device of claim 13 , wherein, for the performing of the summation operation, the one or more processors are further configured to perform the summation operation between the reference element and each of a plurality of cascading summation results determined by performing a cascading summation operation on each of the convolution operation results.

16. The device of claim 15 , wherein
the performing of the cascading summation operation on one of the convolution operation results comprises summing the one of the convolution operation results and other ones of the convolution operation results,
wherein the other ones of the convolution operation results correspond to one or more differential windows preceding, in the sliding direction, a differential window corresponding to the one of the convolution operation results.

17. The device of claim 12 , wherein the one or more processors are further configured to convert, in response to each of the differential windows being a bit data format, each of the differential windows into a data format comprising information about a significant bit digit representing a bit value of 1,
wherein the convolution operation results between the kernel and each of the differential windows are convolution operation results between the kernel and each of the differential windows converted into the data format.

18. The device of c

Assignees

• Samsung Electronics Co Ltd

Inventors

• Mahmoud Mostafa
• Moshovos Andreas

Classifications

• G06N3/0495

  Quantised networks; Sparse networks; Compressed networks · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06V10/955 Primary

  using specific electronic processors · CPC title

• G06F7/533

  Reduction of the number of iteration steps or stages, e.g. using the Booth algorithm, log-sum, odd-even · CPC title

• G06F18/2163

  Partitioning the feature space · CPC title