Adaptive image enhancement using inter-channel correlation information

Patent metadata

Field	Value
Publication number	US-12047613-B2
Application number	US-202218064745-A
Country	US
Kind code	B2
Filing date	Dec 12, 2022
Priority date	Jun 10, 2020
Publication date	Jul 23, 2024
Grant date	Jul 23, 2024

Abstract

The present disclosure relates to image modification such as an image enhancement. The image enhancement may be applied for any image modification and it may be applied during or after image encoding and/or decoding, e.g. as a loop filter or a post filter. In particular, the image modification includes a multi-channel processing in which a primary channel is processed separately and secondary channels are processed based on the processed primary channel. The processing is based on a neural network. In order to enhance the image modification performance, prior to applying the modification, the image channels are analyzed and a primary channel and the secondary channels are determined, which may vary for multiples of images, images or image areas.

First claim

What is claimed is:

1. A method for modifying an image region represented by two or more image channels, the method comprising:
selecting one of the two or more image channels as a primary channel and another at least one of the two or more image channels as a secondary channel;
processing the primary channel with a first neural network to obtain a modified primary channel;
processing the secondary channel with a second neural network to obtain a modified secondary channel, wherein the processing with the second neural network is based on the modified primary channel;
obtaining a modified image region based on the modified primary channel and the modified secondary channel; and
rearranging pixels of each of the at least two image channels of the image region into a plurality, S, of sub-regions wherein:
each of the sub-regions of an image channel among the at least two image channels contains a subset of samples of the image channel,
for all image channels, the horizontal dimensions of the sub-regions are the same and equal to an integer multiple mh of the greatest common divisor of the horizontal dimension of the image across all image channels, and
for all image channels, the vertical dimensions of the sub-regions are the same and equal to an integer multiple my of the greatest common divisor of the vertical dimension of the image across all image channels.

2. The method according to claim 1 , wherein the step of selecting the primary channel and the secondary channel among the two or more image channels is performed based on an output of a classifier based on a neural network, to which the two or more image channels are inputted.

3. The method according to claim 1 , wherein the two or more image channels include a color channel or a feature channel.

4. The method according to claim 1 , wherein the image region is one of the following:
a patch of a predetermined size corresponding to a part of an image or a part of a plurality of images, or
an image or a plurality of images.

5. The method according to claim 1 , comprising choosing a minimum size for the image region based on the number of hidden layers of each of the first neural network and the second neural network, wherein the minimum size is the smaller of at least 2*((kernel_size−1)/2*n_layers)+1 for each of the respective first or second neural network, with kernel_size being the size of the kernel of the respective neural network which is a convolutional neural network and n_layers being the number of the layers of the respective neural network.

6. The method according to claim 1 , wherein
the S sub-regions of the image region are disjoint with S=mh*mv, and have horizontal dimension dimh and vertical dimension dimv, and
a sub-region includes samples of the image region on the positions {kh*mh+offh, kv*mv+offv}, with kh ∈ [0, dimh−1] and kv ∈ [0, dimv−1], wherein
each combination of offh and offv specifies the respective sub-region with offk ∈ [1, mh] and offv ∈ [1, mv].

7. A method for encoding an image or a video sequence of images, comprising:
obtaining an original image region;
encoding the obtained image region into a bitstream, wherein the encoded image region is represented by two or more image channels; and
modifying an image region obtained by reconstructing the encoded image region by performing the following steps:
selecting one of the two or more image channels as a primary channel and another at least one of the two or more image channels as a secondary channel;
processing the primary channel with a first neural network to obtain a modified primary channel;
processing the secondary channel with a second neural network to obtain a modified secondary channel, wherein the processing with the second neural network is based on the modified primary channel;
obtaining a modified image region based on the modified primary channel and the modified secondary channel; and
rearranging pixels of each of the at least two image channels of the image region into a plurality, S, of sub-regions wherein:
each of the sub-regions of an image channel among the at least two image channels contains a subset of samples of the image channel,
for all image channels, the horizontal dimensions of the sub-regions are the same and equal to an integer multiple mh of the greatest common divisor of the horizontal dimension of the image across all image channels, and
for all image channels, the vertical dimensions of the sub-regions are the same and equal to an integer multiple my of the greatest common divisor of the vertical dimension of the image across all image channels.

8. The method for encoding an image or a video sequence of images according to claim 7 , comprising a step of including into the bitstream an indication of the selected primary channel.

9. The method for encoding an image or a video sequence of images according to claim 7 comprising:
obtaining a plurality of image regions;
applying the method for modifying the obtained image region to the image regions of the obtained plurality of image regions individually; and
including into the bitstream for each of the plurality of image regions at least one of:
an indication indicating that the method for modifying the obtained image region is not to be applied for the image region, or
an indication of the selected primary channel for the image region.

10. The method for encoding an image or a video sequence of images according to claim 7 , wherein, the selection of the primary channel and the secondary channel is performed based on the reconstructed image region without referring to the obtained image region input to the encoding step.

11. The method for encoding an image or a video sequence of images according to claim 7 , wherein the two or more image channels include a color channel or a feature channel.

12. The method for encoding an image or a video sequence of images according to claim 7 , further comprising: choosing a minimum size for the image region based on the number of hidden layers of each of the first neural network and the second neural network, wherein the minimum size is the smaller of at least 2*((kernel_size−1)/2*n_layers)+1 for each of the respective first or second neural network, with kernel_size being the size of the kernel of the respective neural network which is a convolutional neural network and n_layers being the number of the layers of the respective neural network.

13. An apparatus for modifying an image region represented by two or more image channels, wherein the apparatus comprising:
at least one processor; and
at least one memory coupled to the at least one processor and storing instructions for execution by the at least one processor to cause the at least one processor to perform the following steps:
selecting one of the two or more image channels as a primary channel and another at least one of the two or more image channels as a secondary channel;
processing the primary channel with a first neural network to obtain a modified primary channel;
processing the secondary channel with a second neural network to obtain a modified secondary channel, wherein the processing with the second neural network is based on the modified primary channel;
obtaining a modified image region based on the modified primary channel and the modified secondary channel; and
rearranging pixels of each of the at least two image channels of the image region into a plurality, S, of sub-regions wherein:
each of the sub-regions of an image channel among the at least two image channels contains a subset of samples of the image channel,
for all image channels, the horizontal dimensions of the sub-regions are th

Assignees

• Huawei Tech Co Ltd

Inventors

• Cui Kai
• Boev Atanas
• Alshina Elena Alexandrovna
• Steinbach Eckehard

Classifications

• H04N19/436

  using parallelised computational arrangements · CPC title

• H04N19/186

  the unit being a colour or a chrominance component · CPC title

• H04N19/17

  the unit being an image region, e.g. an object · CPC title

• H04N19/167

  Position within a video image, e.g. region of interest [ROI] · CPC title

• H04N19/136

  Incoming video signal characteristics or properties · CPC title