Joint-component neural network based filtering during video coding

Patent metadata

Field	Value
Publication number	US-11825101-B2
Application number	US-202117493543-A
Country	US
Kind code	B2
Filing date	Oct 4, 2021
Priority date	Oct 5, 2020
Publication date	Nov 21, 2023
Grant date	Nov 21, 2023

Abstract

An example device for decoding video data includes a memory configured to store video data; and one or more processors implemented in circuitry and configured to: apply a downsampling convolutional neural network layer to a first color component of a block of video data, the first color component of the block having a first size, wherein applying the downsampling convolutional neural network layer to the first color component generates a downsampled first color component having a second size smaller than the first size; filter a second color component having the second size to form a filtered second color component; concatenate the downsampled first color component with the filtered second color component to form concatenated color components; and filter the concatenated color components to form a filtered concatenated component including a filtered downsampled first color component.

First claim

What is claimed is:

1. A method of filtering decoded video data, the method comprising:
applying a downsampling first convolutional neural network layer to a first color component of a block of video data, the first color component of the block having a first size, wherein applying the downsampling convolutional neural network layer to the first color component generates a downsampled first color component having a second size smaller than the first size;
filtering a second color component having the second size to form a filtered second color component using a second convolutional neural network layer, the second convolutional neural network layer layer being different from the first convolutional neural network layer;
wherein the first color component is a luminance component, and wherein the second color component is one of a blue hue chrominance component or a red hue chroma chrominance component;
concatenating the downsampled first color component with the filtered second color component to form concatenated color components; and
filtering the concatenated color components to form a filtered concatenated component including a filtered downsampled first color component using at least a third convolutional neural network layer.

2. The method of claim 1 , further comprising upsampling the filtered downsampled first color component to the first size.

3. The method of claim 1 , further comprising combining two or more filtered downsampled blocks of the first color component, including the filtered downsampled first color component, to generate an upsampled first color component having the first size.

4. The method of claim 3 , wherein the first size comprises 2N×2N, and wherein the two or more filtered downsampled first color components of the first color component comprise four N×N filtered downsampled blocks of the first color component.

5. The method of claim 1 , wherein the downsampling convolutional neural network layer comprises a 3×3×M convolutional neural network layer filter with a stride of 2.

6. The method of claim 1 , wherein the convolutional neural network layer comprises a residual processing unit.

7. The method of claim 6 , wherein the residual processing unit comprises a first 3×3×K×K convolution layer, a PReLU layer, and a second 3×3×K×K convolution layer.

8. The method of claim 1 , further comprising filtering a third color component of the block of video data using the convolutional neural network layer.

9. The method of claim 8 , wherein the second size comprises the smaller of a size of the second color component or a size of the third color component.

10. A device for decoding video data, the device comprising:
a memory configured to store video data; and
one or more processors implemented in circuitry and configured to:
apply a downsampling first convolutional neural network layer to a first color component of a block of video data, the first color component of the block having a first size, wherein applying the downsampling convolutional neural network layer to the first color component generates a downsampled first color component having a second size smaller than the first size;
filter a second color component having the second size to form a filtered second color component using a second convolutional neural network layer, the second convolutional neural network layer layer being different from the first convolutional neural network layer;
wherein the first color component is a luminance component, and wherein the second color component is one of a blue hue chrominance component or a red hue chroma chrominance component;
concatenate the downsampled first color component with the filtered second color component to form concatenated color components; and
filter the concatenated color components to form a concatenated color component including a filtered downsampled first color component using at least a third convolutional neural network layer.

11. The device of claim 10 , wherein the one or more processors are further configured to upsample the filtered downsampled first color component to the first size.

12. The device of claim 10 , wherein the one or more processors are further configured to combine two or more filtered downsampled blocks of the first color components, including the filtered downsampled first color component, to generate an upsampled first color component having the first size.

13. The device of claim 12 , wherein the first size comprises 2N×2N, and wherein the two or more filtered downsampled blocks of the first color component comprise four N×N filtered downsampled blocks of the first color component.

14. The device of claim 10 , wherein the downsampling convolutional neural network layer comprises a 3×3×M convolutional neural network layer filter with a stride of 2.

15. The device of claim 10 , wherein the convolutional neural network layer comprise a residual processing unit.

16. The device of claim 15 , wherein the residual processing unit comprises a first 3×3×K×K convolution layer, a PReLU layer, and a second 3×3×K×K convolution layer.

17. The device of claim 10 , wherein the one or more processors are further configured to filter a third color component of the block of video data using the convolutional neural network layer.

18. The device of claim 17 , wherein the second size comprises the smaller of a size of the second color component or a size of the third color component.

19. The device of claim 10 , further comprising a display configured to display video data corresponding to the concatenated color components.

20. The device of claim 10 , wherein the device comprises one or more of a camera, a computer, a mobile device, a broadcast receiver device, or a set-top box.

21. A non-transitory computer-readable storage medium having stored thereon instructions that, when executed, cause a processor to:
apply a downsampling first convolutional neural network layer to a first color component of a block of video data, the first color component of the block having a first size, wherein applying the downsampling convolutional neural network layer to the first color component generates a downsampled first color component having a second size smaller than the first size;
filter a second color component having the second size to form a filtered second color component using a second convolutional neural network layer, the second convolutional neural network layer layer being different from the first convolutional neural network layer;
wherein the first color component is a luminance component, and wherein the second color component is one of a blue hue chrominance component or a red hue chroma chrominance component;
concatenate the downsampled first color component with the filtered second color component to form concatenated color components; and
filter the concatenated color components to form a filtered concatenated component including a filtered downsampled first color component using at least a third convolutional neural network layer.

22. The computer-readable storage medium of claim 21 , further comprising instructions that cause the processor to upsample the filtered downsampled first color component to the first size.

23. The computer-readable storage medium of claim 21 , further comprising instructions that cause the processor to combine two or more filtered downsampled blocks of the first color component, including the filtered downsampled first color component, to generate an upsampled firs

Assignees

• Qualcomm Inc

Inventors

• Chen Jianle
• Wang Hongtao
• Kotra Venkata Meher Satchit Anand
• Karczewicz Marta

Classifications

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• H04N19/186 Primary

  the unit being a colour or a chrominance component · CPC title

• G06N3/04

  Architecture, e.g. interconnection topology · CPC title

• G06N3/10

  Interfaces, programming languages or software development kits, e.g. for simulating neural networks · CPC title

• H04N19/117 Primary

  Filters, e.g. for pre-processing or post-processing (sub-band filter banks H04N19/635) · CPC title