Lossy image compression with diffusion models

Patent metadata

Field	Value
Publication number	US-2025157087-A1
Application number	US-202418920557-A
Country	US
Kind code	A1
Filing date	Oct 18, 2024
Priority date	Nov 15, 2023
Publication date	May 15, 2025
Grant date	—

Abstract

In some embodiments, a method receives a quantized latent representation of an image in a latent space. The image is encoded into a representation in the latent space and quantized to generate the quantized latent representation. A time step parameter is received that is generated based on the representation. The method performs an inverse quantization process to generate a reconstructed representation. A diffusion model performs a denoising process for a number of iterations based on the time step parameter to remove noise from the reconstructed representation to generate a denoised reconstructed representation. The denoised reconstructed representation is decoded into a reconstructed image.

First claim

What is claimed is:

1 . A method comprising:
receiving a quantized latent representation of an image in a latent space, wherein the image is encoded into a latent representation in the latent space and quantized to generate the quantized latent representation; receiving a time step parameter that is generated based on the latent representation; performing an inverse quantization process to generate a reconstructed latent representation; performing, using a diffusion model, a denoising process for a number of iterations based on the time step parameter to remove noise from the reconstructed latent representation to generate a denoised reconstructed latent representation; and decoding the denoised reconstructed latent representation into a reconstructed image.

2 . The method of claim 1 , wherein:
the quantized latent representation is entropy coded, and the quantized latent representation is entropy decoded before performing the inverse quantization process.

3 . The method of claim 1 , further comprising:
receiving a quantization setting that is generated based on the latent representation in the latent space; and performing the inverse quantization process using the quantization setting.

4 . The method of claim 3 , wherein the quantization setting is used by the inverse quantization process to adjust a number of bits that are used for quantization to generate the reconstructed latent representation.

5 . The method of claim 3 , wherein:
the quantization setting is generated based on the latent representation of the image and an input parameter, wherein the input parameter is based on a rate distortion balance.

6 . The method of claim 1 , wherein:
the time step parameter is generated based on the latent representation of the image and an input parameter, wherein the input parameter is based on a rate distortion balance.

7 . The method of claim 1 , wherein:
a quantization process and inverse quantization process add quantization error to the reconstructed latent representation compared to the latent representation of the image, and the diffusion model is configured to remove noise associated with the quantization error.

8 . The method of claim 1 , wherein:
a quantization error from generating the quantized latent representation adds noise to the reconstructed latent representation, and the diffusion model denoises the reconstructed latent representation to remove noise from the reconstructed latent representation.

9 . The method of claim 1 , wherein:
a network is trained to generate the time step parameter.

10 . The method of claim 9 , wherein:
the diffusion model is not trained during a training of the network.

11 . The method of claim 9 , wherein:
the network is trained to generate a quantization setting that is used by the inverse quantization process to adjust a number of bits that are used for quantization to generate the reconstructed latent representation.

12 . A non-transitory computer-readable storage medium having stored thereon computer executable instructions, which when executed by a computing device, cause the computing device to be operable for:
receiving a quantized latent representation of an image in a latent space, wherein the image is encoded into a latent representation in the latent space and quantized to generate the quantized latent representation; receiving a time step parameter that is generated based on the latent representation; performing an inverse quantization process to generate a reconstructed latent representation; performing, using a diffusion model, a denoising process for a number of iterations based on the time step parameter to remove noise from the reconstructed latent representation to generate a denoised reconstructed latent representation; and decoding the denoised reconstructed latent representation into a reconstructed image.

13 . A method comprising:
receiving an image; encoding the image into a latent representation in a latent space; estimating a time step parameter based on the latent representation; performing a quantization process on the latent representation to generate a quantized latent representation; and transmitting the quantized latent representation to a receiver, wherein an inverse quantization process is performed to generate a reconstructed latent representation and a diffusion model performs a denoising process for a number of iterations based on the time step parameter to remove noise from the reconstructed latent representation.

14 . The method of claim 13 , wherein:
the quantized latent representation is entropy coded, and the quantized latent representation is entropy decoded before performing the inverse quantization process.

15 . The method of claim 13 , further comprising:
determining a quantization setting that is generated based on the latent representation in the latent space; and performing the quantization process using the quantization setting.

16 . The method of claim 15 , wherein:
the quantization setting is generated based on the latent representation of the image and an input parameter, wherein the input parameter is based on a rate distortion tradeoff.

17 . The method of claim 13 , wherein estimating the time step parameter comprises:
estimating the time step parameter based on the latent representation of the image and an input parameter, wherein the input parameter is based on a rate distortion tradeoff.

18 . The method of claim 13 , wherein:
a quantization error from the quantization process adds noise to the reconstructed latent representation, and the diffusion model denoises the reconstructed latent representation to remove noise from the reconstructed latent representation.

19 . The method of claim 13 , wherein:
a network is trained to generate the time step parameter.

20 . The method of claim 9 , wherein:
the network is trained to generate a quantization setting that is used by the quantization process to adjust a number of bits that are used for quantization to generate the quantized latent representation.

Assignees

• Disney Entpr Inc
• Eth Zuerich Eidgenoessische Technische Hochschule Zuerich

Inventors

• Relic Lucas
• De Albuquerque Azevedo Roberto Gerson
• Schroers Christopher Richard
• Xue Yuanyi
• Labrozzi Scott

Classifications

• H04N19/86

  involving reduction of coding artifacts, e.g. of blockiness · CPC title

• H04N19/44

  Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder · CPC title

• H04N19/124 Primary

  Quantisation · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T2207/20081

  Training; Learning · CPC title