Systems and methods for generating a latent space residual

Patent metadata

Field	Value
Publication number	US-11012718-B2
Application number	US-201916557920-A
Country	US
Kind code	B2
Filing date	Aug 30, 2019
Priority date	Aug 30, 2019
Publication date	May 18, 2021
Grant date	May 18, 2021

Abstract

Systems and methods are disclosed for generating a latent space residual. A computer-implemented method may use a computer system that includes non-transient electronic storage, a graphical user interface, and one or more physical computer processors. The computer-implemented method may include: obtaining a target frame, obtaining a reconstructed frame, encoding the target frame into a latent space to generate a latent space target frame, encoding the reconstructed frame into the latent space to generate a latent space reconstructed frame, and generating a latent space residual based on the latent space target frame and the latent space reconstructed frame.

First claim

What is claimed is:

1. A computer-implemented method comprising:
obtaining, from a non-transient electronic storage, a target frame;
obtaining, from the non-transient electronic storage, a reconstructed frame, wherein the reconstructed frame is based on surrounding reference frames;
encoding, with a physical computer processor, the target frame into a latent space to generate a latent space target frame;
encoding, with the physical computer processor, the reconstructed frame into the latent space to generate a latent space reconstructed frame; and
generating, with the physical computer processor, a latent space residual based on the latent space target frame and the latent space reconstructed frame.

2. The computer-implemented method of claim 1 , further comprising decoding, with the physical computer processor, the latent space residual and the latent space reconstructed frame to generate a decoded target frame.

3. The computer-implemented method of claim 1 , wherein the reconstructed frame is generated by:
obtaining, from the non-transient electronic storage, the surrounding reference frames;
encoding, with the physical computer processor, the surrounding reference frames;
decoding, with the physical computer processor, the surrounding reference frames to generate one or more decoded reference frames; and
predicting, with the physical computer processor, the reconstructed frame based on the one or more decoded reference frames.

4. The computer-implemented method of claim 1 , wherein encoding the target frame and the reconstructed frame maps the target frame and the reconstructed frame from an image space to the latent space.

5. The computer-implemented method of claim 1 , wherein the latent space residual and the latent space reconstructed frame are quantized in the latent space.

6. The computer-implemented method of claim 1 , wherein the latent space residual and the latent space reconstructed frame are entropy coded.

7. A computer-implemented method comprising:
obtaining, from a non-transient electronic storage, a target frame;
obtaining, from the non-transient electronic storage, one or more reference frames surrounding the target frame;
obtaining, from the non-transient electronic storage, an encoder and a decoder;
applying, with a physical computer processor, the one or more reference frames to the decoder to generate one or more decoded reference frames;
predicting, with the physical computer processor, a reconstructed frame corresponding to the target frame based on the one or more decoded reference frames,
applying, with the physical computer processor, the target frame to the encoder to generate a latent space target frame;
applying, with the physical computer processor, the reconstructed frame to the encoder to generate a latent space reconstructed frame; and
generating, with the physical computer processor, a latent space residual based on the latent space target frame and the latent space reconstructed frame.

8. The computer-implemented method of claim 7 , further comprising applying, with the physical computer processor, the latent space residual and the latent space reconstructed frame to the decoder to generate a decoded target frame.

9. The computer-implemented method of claim 7 , wherein the encoder maps an image space to the latent space.

10. The computer-implemented method of claim 7 , wherein the decoder maps the latent space to an image space.

11. The computer-implemented method of claim 7 , wherein obtaining the encoder and the decoder comprises obtaining, from the non-transient electronic storage, an image transformative model, wherein the image transformative model comprises the encoder and the decoder, and wherein the image transformative model is based on a neural network.

12. The computer-implemented method of claim 7 , wherein the latent space residual and the latent space reconstructed frame are quantized in the latent space.

13. The computer-implemented method of claim 7 , wherein the latent space residual and the latent space reconstructed frame are entropy coded.

14. A system for generating a latent space residual, the system comprising:
non-transient electronic storage;
a physical computer processor configured by machine-readable instructions to:
obtain a target frame;
obtain a reconstructed frame, wherein the reconstructed frame is based on surrounding reference frames;
encode the target frame into a latent space to generate a latent space target frame;
encode the reconstructed frame into the latent space to generate a latent space reconstructed frame; and
generate a latent space residual based on the latent space target frame and the latent space reconstructed frame.

15. The system of claim 14 , wherein the physical computer processor is further configured by machine-readable instructions to decode the latent space residual and the latent space reconstructed frame to generate a decoded target frame.

16. The system of claim 15 , wherein the physical computer processor is further configured by machine-readable instructions to display, via a graphical user interface, the decoded target frame.

17. The system of claim 14 , wherein the physical computer processor is further configured by machine-readable instructions to:
obtain the surrounding reference frames;
encode the surrounding reference frames;
decode the surrounding frames to generate one or more decoded reference frames; and
predict the reconstructed frame based on the one or more decoded reference frames.

18. The system of claim 14 , wherein encoding the target frame and the reconstructed frame maps the target frame and the reconstructed frame from an image space to the latent space.

19. The system of claim 14 , wherein the latent space residual and the latent space reconstructed frame are quantized in the latent space.

20. The system of claim 14 , wherein the latent space residual and the latent space reconstructed frame are entropy coded.

Assignees

• Disney Entpr Inc

Inventors

• Schroers Christopher
• Campos Joaquim
• Djelouah Abdelaziz
• Xue Yuanyi
• Doggett Erika Varis
• Mcphillen Jared
• Labrozzi Scott

Classifications

• H04N19/91 Primary

  Entropy coding, e.g. variable length coding [VLC] or arithmetic coding · CPC title

• H04N19/50 Primary

  using predictive coding (H04N19/61 takes precedence) · CPC title

• G06V10/82

  using neural networks · CPC title

• G06V10/454

  Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN] · CPC title

• G06N3/045

  Combinations of networks · CPC title