System, method, and computer program for content adaptive online training for multiple blocks in neural image compression

Patent metadata

Field	Value
Publication number	US-12347149-B2
Application number	US-202217950569-A
Country	US
Kind code	B2
Filing date	Sep 22, 2022
Priority date	Dec 13, 2021
Publication date	Jul 1, 2025
Grant date	Jul 1, 2025

Abstract

Content-adaptive online training for end-to-end (E2E) neural image compression (NIC) using a neural network performed by at least one processor, is provided, including receiving an input image, to an E2E NIC framework, including one or more blocks, preprocessing a first neural network of the E2E NIC framework, based on the one or more blocks, computing updated parameters using the preprocessed first neural network, encoding the one or more blocks and the updated parameters, updating the first neural network based on the encoded updated parameters, and generating a compressed representation of the encoded one or more blocks using the updated first neural network.

First claim

What is claims is:

1. A method of content-adaptive online training for end-to-end (E2E) neural image compression (NIC) using a neural network performed by at least one processor, the method comprising:
receiving, by an E2E NIC framework, an input image including one or more blocks;
preprocessing a first neural network of the E2E NIC framework, based on the one or more blocks;
computing updated parameters using the preprocessed first neural network, wherein the updated parameters include a learning rate and a number of steps, and wherein the learning rate and the number of steps are selected based on characteristics of the input image;
encoding the one or more blocks and the updated parameters;
updating the first neural network based on the encoded updated parameters; and
generating a compressed representation of the encoded one or more blocks using the updated first neural network.

2. The method according to claim 1 , further comprising:
splitting the input image into the one or more blocks; and
compressing the one or more blocks individually.

3. The method according to claim 1 , further comprising;
decoding the compressed representation using arithmetic decoding; and
generating a reconstructed image based on the decoded compressed representation using a second neural network.

4. The method according to claim 1 , further comprising compressing the updated parameters.

5. The method according to claim 1 , wherein the characteristics of the input image are one of a RGB variance of the input image and an RD performance of the input image.

6. The method according to claim 1 , wherein when preprocessing the first neural network, the first neural network is fine-tuned using the one or more blocks.

7. An apparatus for content-adaptive online training for end-to-end (E2E) neural image compression (NIC) using a neural network, the apparatus comprising:
at least one memory configured to store computer program code; and
at least one processor configured to read the computer program code and operate as instructed by the computer program code, the computer program code including:
receiving code configured to cause the at least one processor to receive, by an E2E NIC framework, an input including one or more blocks;
preprocessing code configured to cause the at least one processor to preprocess a first neural network of the E2E NIC framework, based on the one or more blocks;
computing code configured to cause the at least one processor to compute updated parameters using the preprocessed first neural network, wherein the updated parameters include a learning rate and a number of steps, and wherein the learning rate and the number of steps are selected based on characteristics of the input image;
encoding code configured to cause the at least one processor to encode the one or more blocks and the updated parameters;
updating code configured to cause the at least one processor to update the first neural network based on the encoded updated parameters; and
first generating code configured to cause the at least one processor to generate a compressed representation of the encoded one or more blocks using the updated first neural network.

8. The apparatus of claim 7 , the computer program code further including:
splitting code configured to cause the at least one processor to split the input image into the one or more blocks; and
compressing code configured to cause the at least one processor to compress the one or more blocks individually.

9. The apparatus of claim 7 , the computer program code further including:
decoding code configured to cause the at least one processor to decode the compressed representation using arithmetic decoding; and
second generating code configured to cause the at least one processor to generate a reconstructed image based on the decoded compressed representation using a second neural network.

10. The apparatus of claim 7 , the computer program code further including compressing code configured to cause the at least one processor to compress the updated parameters.

11. The apparatus of claim 7 , wherein the characteristics of the input image are one of a RGB variance of the input image and an RD performance of the input image.

12. The apparatus of claim 7 , wherein when preprocessing the first neural network, the first neural network is fine-tuned using the one or more blocks.

13. A non-transitory computer-readable medium storing instructions that, when executed by at least one processor of an apparatus for content-adaptive online training for end-to-end (E2E) neural image compression (NIC) using a neural network, cause the at least one processor to:
receive, by an E2E NIC framework, an input image including one or more blocks;
preprocess a first neural network of the E2E NIC framework, based on the one or more blocks;
compute updated parameters using the preprocessed first neural network;
encode the one or more blocks and the updated parameters, wherein the updated parameters include a learning rate and a number of steps, and wherein the learning rate and the number of steps are selected based on characteristics of the input image;
update the first neural network based on the encoded updated parameters; and
generate a compressed representation of the encoded one or more blocks using the updated first neural network.

14. The non-transitory computer-readable medium of claim 13 , wherein the instructions further cause the at least one processor to:
split the input image into the one or more blocks; and
compress the one or more blocks individually.

15. The non-transitory computer-readable medium of claim 13 , wherein the instructions further cause the at least one processor to:
decode the compressed representation using arithmetic decoding; and
generate a reconstructed image based on the decoded compressed representation using a second neural network.

16. The non-transitory computer-readable medium of claim 13 , wherein the instructions further cause the at least one processor to compress the updated parameters.

17. The non-transitory computer-readable medium of claim 13 , wherein the characteristics of the input image are one of a RGB variance of the input image and an RD performance of the input image.

18. The non-transitory computer-readable medium of claim 13 , wherein when preprocessing the first neural network, the first neural network is fine-tuned using the one or more blocks.

Assignees

• Tencent America LLC

Inventors

• Ding Ding
• Wang Wei
• Liu Shan

Classifications

• G06N3/045

  Combinations of networks · CPC title

• G06T3/4046

  using neural networks · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T9/002 Primary

  using neural networks · CPC title

• H04N19/176

  the region being a block, e.g. a macroblock · CPC title