Systems and methods for reconstructing frames

Patent metadata

Field	Value
Publication number	US-10972749-B2
Application number	US-201916556083-A
Country	US
Kind code	B2
Filing date	Aug 29, 2019
Priority date	Aug 29, 2019
Publication date	Apr 6, 2021
Grant date	Apr 6, 2021

Abstract

Systems and methods are disclosed for reconstructing a frame. A computer-implemented method may use a computer system that includes non-transient electronic storage, a graphical user interface, and one or more physical computer processors. The computer-implemented method may include: obtaining one or more reference frames from non-transient electronic storage, generating one or more displacement maps based on the one or more reference frames and a target frame with the physical computer processor, generating one or more warped frames based on the one or more reference frames and the one or more displacement maps with the physical computer processor, obtaining a conditioned reconstruction model from the non-transient electronic storage, and generating one or more blending coefficients and one or more reconstructed displacement maps by applying the one or more displacement maps, the one or more warped frames, and a target frame to the conditioned reconstruction model with the physical computer processor.

First claim

What is claimed is:

1. A computer-implemented method comprising:
generating, using an optical flow model, one or more displacement maps based on one or more reference frames and a target frame;
generating one or more warped frames based on the one or more reference frames and the one or more displacement maps;
generating a conditioned reconstruction model by training an initial reconstruction model using training content and one or more reconstruction parameters, wherein the training content comprising a training target frame and one or more training reference frames, and wherein the conditioned reconstruction model optimizes for the one or more reconstruction parameters; and
generating, using the conditioned reconstruction model, one or more blending coefficients and one or more reconstructed displacement maps based on the one or more displacement maps, the one or more warped frames, and the target frame.

2. The computer-implemented method of claim 1 , further comprising
generating a reconstructed target frame using one or more reconstructed reference frames, the one or more blending coefficients, and the one or more reconstructed displacement maps.

3. The computer-implemented method of claim 2 , wherein of the one or more blending coefficients indicate which pixels to use from of the one or more reconstructed reference frames.

4. The computer-implemented method of claim 1 , wherein of the one or more reference frames comprise reconstructed target frames.

5. The computer-implemented method of claim 1 , wherein of the one or more reference frames are separated from the target frame by an interval.

6. The computer-implemented method of claim 5 , wherein the interval is between one frame and five frames.

7. The computer-implemented method of claim 1 , wherein the one or more displacement maps represent motion data based on a difference between the target frame and of the one or more reference frames.

8. The computer-implemented method of claim 1 , wherein of the one or more warped frames are generated by applying of the one or more displacement maps to the one or more reference frames.

9. A computer-implemented method comprising:
obtaining training content, wherein the training content comprises a training target frame and one or more training reference frames, wherein the training target frame and the one or more training reference frames are used to generate one or more corresponding training displacement maps and one or more corresponding training warped frames; and
generating a conditioned reconstruction model by training an initial reconstruction model using the training content and one or more reconstruction parameters, wherein the conditioned reconstruction model optimizes for the one or more reconstruction parameters.

10. The computer-implemented method of claim 9 , further comprising:
generating, using an optical flow model, one or more displacement maps based on one or more reference frames and a target frame;
generating one or more warped frames based on the one or more reference frames and the one or more displacement maps; and
generating, using the conditioned reconstruction model, one or more blending coefficients and one or more reconstructed displacement maps based on the one or more displacement maps, the one or more warped frames, and the target frame.

11. The computer-implemented method of claim 10 , further comprising:
generating a reconstructed target frame using one or more reconstructed reference frames, the one or more blending coefficients, and the one or more reconstructed displacement maps.

12. The computer-implemented method of claim 11 , wherein the one or more blending coefficients indicate which pixels to use from the one or more reconstructed reference frames.

13. The computer-implemented method of claim 10 , wherein the one or more displacement maps represent motion data based on a difference between the target frame the one or more reference frames.

14. A system comprising:
a memory storing one or more instructions; and
one or more processors that execute the one or more instructions to perform the steps of:
generating a conditioned reconstruction model by training an initial reconstruction model using training content and one or more reconstruction parameters, the training content comprising a training target frame and one or more target reference frames, wherein the conditioned reconstruction model optimizes for the one or more reconstruction parameters; and
generating, using the conditioned reconstruction model, one or more blending coefficients and one or more reconstructed displacement maps by applying target content to the conditioned reconstruction model.

15. The system of claim 14 , wherein the one or more processors are further configured to perform the steps of:
generating a reconstructed target frame using one or more reconstructed reference frames, the one or more blending coefficients, and the one or more reconstructed displacement maps.

16. The system of claim 15 , wherein the one or more blending coefficients indicate which pixels to use from the one or more reconstructed reference frames.

17. The system of claim 14 , wherein the target content comprises a target frame, one or more displacement maps, and one or more corresponding warped frames.

18. The system of claim 17 , wherein the one or more displacement maps represent motion data based on a difference between the target frame and one or more reference frames.

19. The system of claim 17 , wherein one or more reference frames are separated from the target frame by one or more intervals.

20. The system of claim 17 , wherein the one or more corresponding warped frames are generated by applying the one or more displacement maps to the target frame.

Assignees

• Disney Entpr Inc

Inventors

• Schroers Christopher
• Campos Joaquim
• Djelouah Abdelaziz
• Xue Yuanyi
• Doggett Erika Varis
• Mcphillen Jared
• Labrozzi Scott

Classifications

• H04N19/20 Primary

  using video object coding · CPC title

• G06V10/62

  relating to a temporal dimension, e.g. time-based feature extraction; Pattern tracking · CPC title

• G06V20/46

  Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames · CPC title

• G06V10/70

  using pattern recognition or machine learning (optical pattern recognition or electronic computations therefor G06V10/88) · CPC title

• G06V10/462

  Salient features, e.g. scale invariant feature transforms [SIFT] · CPC title