Method and device for processing multi-channel feature map images

Patent metadata

Field	Value
Publication number	US-10733767-B2
Application number	US-201815967039-A
Country	US
Kind code	B2
Filing date	Apr 30, 2018
Priority date	May 31, 2017
Publication date	Aug 4, 2020
Grant date	Aug 4, 2020

Abstract

A convolutional neural network-based image processing method is provided. The method includes: receiving, in a second layer, multi-channel feature map images generated by applying a convolution operation to an input image of a convolutional neural network having a plurality of layers with a plurality of filter kernels of a first layer; analyzing a dynamic range of the multi-channel feature map images; re-ordering the multi-channel feature map images, based on the dynamic range; and processing the re-ordered multi-channel feature map images in the second layer.

First claim

What is claimed is:

1. A convolutional neural network-based image processing method comprising:
receiving, in a second layer, multi-channel feature map images generated by applying a convolution operation to an input image of a convolutional neural network having a plurality of layers with a plurality of filter kernels of a first layer;
analyzing a dynamic range of the multi-channel feature map images;
re-ordering the multi-channel feature map images, based on the dynamic range; and
processing the re-ordered multi-channel feature map images in the second layer.

2. The convolutional neural network-based image processing method of claim 1 , wherein the analyzing comprises obtaining the dynamic range for each of the multi-channel feature map images, based on a maximum output value and a minimum output value of each of the plurality of filter kernels of the first layer.

3. The convolutional neural network-based image processing method of claim 1 , wherein the re-ordering comprises aligning the multi-channel feature map images in a descending order or an ascending order in a channel direction, based on the dynamic range.

4. The convolutional neural network-based image processing method of claim 1 , wherein the processing comprises performing inter-channel inter-prediction on the re-ordered multi-channel feature map images.

5. The convolutional neural network-based image processing method of claim 1 , wherein the processing comprises performing encoding and decoding operations on the re-ordered multi-channel feature map images,
wherein the encoding and decoding operations in the second layer are performed according to a pipelined process with a convolution operation in a layer subsequent to the second layer among the plurality of layers.

6. The convolutional neural network-based image processing method of claim 1 , wherein the processing comprises determining a coding unit having a pixel depth in a channel direction of the multi-channel feature map images, the pixel depth corresponding to a number of channels in the channel direction in the multi-channel feature map images.

7. The convolutional neural network-based image processing method of claim 6 , wherein the coding unit corresponds to a basic unit of a filter kernel operation in a layer subsequent to the second layer among the plurality of layers.

8. The convolutional neural network-based image processing method of claim 6 , wherein a size of the coding unit is X×Y×Z, where X, Y, and Z are integers.

9. The convolutional neural network-based image processing method of claim 6 , wherein the processing further comprises:
dividing the coding unit into sub-coding units in the channel direction; and
performing encoding on each of the sub-coding units.

10. The convolutional neural network-based image processing method of claim 9 , wherein the performing of encoding on each of the sub-coding units comprises:
encoding information on whether an encoding mode of a current sub-coding unit is equal to an encoding mode of a previous sub-coding unit; and
encoding information on the encoding mode of the current sub-coding unit when the encoding mode of the current sub-coding unit is not equal to the encoding mode of the previous sub-coding unit.

11. A convolutional neural network-based image processing device comprises:
a storage; and
a controller configured to control the image processing device to receive, in a second layer, multi-channel feature map images generated by applying a convolution operation to an input image of a convolutional neural network having a plurality of layers with a plurality of filter kernels of a first layer, to analyze a dynamic range of the multi-channel feature map images, to re-order the multi-channel feature map images, based on the dynamic range, to process the re-ordered multi-channel feature map images in the second layer, and to store the processed re-ordered multi-channel feature map images in the storage.

12. The convolutional neural network-based image processing device of claim 11 , wherein the controller is further configured to control the image processing device to obtain the dynamic range for each of the multi-channel feature map images, based on a maximum output value and a minimum output value of each of the plurality of filter kernels of the first layer.

13. The convolutional neural network-based image processing device of claim 11 , wherein the controller is further configured to control the image processing device to align the multi-channel feature map images in a descending order or an ascending order in a channel direction, based on the dynamic range.

14. The convolutional neural network-based image processing device of claim 11 , wherein the controller is further configured to control the image processing device to perform inter-channel inter-prediction on the re-ordered multi-channel feature map images.

15. The convolutional neural network-based image processing device of claim 11 , wherein the controller is further configured to control the image processing device to perform encoding and decoding operations on the re-ordered multi-channel feature map images,
wherein the controller is further configured to control the image processing device to perform the encoding and decoding operations in the second layer according to a pipelined process with a convolution operation in a layer subsequent to the second layer among the plurality of layers.

16. The convolutional neural network-based image processing device of claim 11 , wherein the controller is further configured to control the image processing device to determine a coding unit having a pixel depth in a channel direction of the multi-channel feature map images, the pixel depth corresponding to a number of channels in the channel direction in the multi-channel feature map images.

17. The convolutional neural network-based image processing device of claim 16 , wherein the coding unit corresponds to a basic unit of a filter kernel operation in a layer subsequent to the second layer among the plurality of layers.

18. The convolutional neural network-based image processing device of claim 16 , wherein the controller is further configured to control the image processing device to control the image processing device to divide the coding unit into sub-coding units in the channel direction, and to perform encoding on each of the sub-coding units.

19. The convolutional neural network-based image processing device of claim 18 , wherein the controller is configured to control the image processing device to encode information on whether an encoding mode of a current sub-coding unit is equal to an encoding mode of a previous sub-coding unit, and to encode information on the encoding mode of the current sub-coding unit when the encoding mode of the current sub-coding unit is not equal to the encoding mode of the previous sub-coding unit.

20. A non-transitory computer-readable recording medium having stored thereon program commands which, when executed, cause a convolution neural network-based image process device to perform:
receiving, in a second layer, multi-channel feature map images generated by applying a convolution operation to an input image of a convolutional neural network having a plurality of layers with a plurality of filter kernels of a first layer;
analyzing a dynamic range of the received multi-channel feature map images;
re-ordering the received multi-channel feature map images based on the dynamic range; and
processing the re-ordered multi-channel feature map images in

Assignees

• Samsung Electronics Co Ltd

Inventors

• Cho Dae-Sung
• Lee Won-Jae

Classifications

• G06V10/82

  using neural networks · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06V10/454

  Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN] · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/0455

  Auto-encoder networks; Encoder-decoder networks · CPC title