Video reconstruction from videos with ultra-low frame-per-second

Patent metadata

Field	Value
Publication number	US-12488481-B2
Application number	US-202217887034-A
Country	US
Kind code	B2
Filing date	Aug 12, 2022
Priority date	Aug 12, 2022
Publication date	Dec 2, 2025
Grant date	Dec 2, 2025

Abstract

In one embodiment, a method includes accessing a video captured by cameras which is associated with a first framerate lower than a threshold framerate, for any two adjacent frames of the accessed video: generating a warped frame from the two adjacent frames based on an optical flow associated with the two adjacent frames, determining alignments for the two adjacent frames, respectively, fusing the determined alignments for the two adjacent frames, and generating a reconstructed frame based on the fused alignment, and reconstructing the accessed video based on the any two adjacent frames and their respective reconstructed frames, wherein the reconstructed video is associated with a second framerate higher than the threshold framerate.

First claim

What is claimed is:

1 . A method comprising:
accessing a video captured by one or more cameras, wherein the accessed video is associated with a first framerate lower than a threshold framerate; for any two adjacent frames of the accessed video:
generating respective warped frames for each frame of the any two adjacent frames based on a respective optical flow associated with each frame of the any two adjacent frames;
fusing the respective warped frames to generate an intermediate frame;
determining respective alignments associated with each frame of the any two adjacent frames based on aligning each of the any two adjacent frames to the intermediate frame;
fusing the respective determined alignments to generate residual information associated with the intermediate frame; and
generating a reconstructed frame based on adding the intermediate frame with the residual information; and
reconstructing the accessed video based on the any two adjacent frames and their respective reconstructed frame, wherein the reconstructed accessed video is associated with a second framerate higher than the threshold framerate.

2 . The method of claim 1 , wherein the accessing the video captured by the one or more cameras comprises: receiving, via a wireless network, the accessed video transmitted from the one or more cameras.

3 . The method of claim 2 , wherein the wireless network is associated with a bandwidth lower than a threshold bandwidth.

4 . The method of claim 1 , wherein the one or more cameras are associated with a wearable device.

5 . The method of claim 1 , further comprising: extracting respective features for the any two adjacent frames, wherein the respective alignments are determined based on the respective extracted features.

6 . The method of claim 5 , wherein the determining the respective alignments for each frame of the any two adjacent frames comprises, for each frame:
determining an offset field by concatenating the respective optical flow to the respective extracted features; generating a refined offset field by adding the determined offset field to a flow field associated with the respective extracted features; determining correspondences between the any two adjacent frames in a forward direction and a backward direction; and convolving locations corresponding to the correspondences based on the refined offset field.

7 . The method of claim 5 , further comprising: generating, based on the respective extracted features for the any two adjacent frames by a warping algorithm, features associated with one or more reconstructed frames.

8 . The method of claim 1 , wherein:
the any two adjacent frames depict one or more moving objects, the determining the respective alignments for the any two adjacent frames comprises: determining one or more respective placements associated with the one or more moving objects in one or more reconstructed frames.

9 . The method of claim 1 , wherein:
the any two adjacent frames are associated with a first movement associated with the one or more cameras, and the determining the respective alignments for the any two adjacent frames comprises: determining a second movement associated with the one or more cameras, wherein the respective determined alignments are based on the second movement.

10 . The method of claim 1 , wherein the fusing the respective determined alignments for the any two adjacent frames is based on contextual information.

11 . The method of claim 1 , further comprising: adding an overlay associated with the any two adjacent frames to the reconstructed frame.

12 . The method of claim 1 , further comprising: adding one of the any two adjacent frames to the reconstructed frame.

13 . The method of claim 1 , wherein:
the any two adjacent frames comprise a first frame and a second frame, the generating the respective warped frame comprises:
determining a forward optical flow from the first frame to the second frame and a backward optical flow from the second frame to the first frame;
determining an intermediate optical flow at a predetermined time interval;
determining, based on splatting, a first reverse optical flow from a reconstructed frame to the first frame and a second reverse optical flow from the second frame to the reconstructed frame; and
warping the first frame to the predetermined time interval based on the first reverse optical flow to generate a warped first frame and warping the second frame to the predetermined time interval based on the second reverse optical flow to generate a warped second frame; and
the fusing the respective warped frame comprises fusing the warped first frame and the warped second frame.

14 . A non-transitory computer-readable medium storing instructions that, when executed, cause:
accessing a video captured by one or more cameras, wherein the accessed video is associated with a first framerate lower than a threshold framerate; for any two adjacent frames of the accessed video:
generating respective warped frames for each frame of the any two adjacent frames based on a respective optical flow associated with each frame of the any two adjacent frames;
fusing the respective warped frames to generate an intermediate frame;
determining respective alignments associated with each frame of the any two adjacent frames based on aligning each of the any two adjacent frames to the intermediate frame;
fusing the respective determined alignments to generate residual information associated with the intermediate frame; and
generating a reconstructed frame based on adding the intermediate frame with the residual information; and
reconstructing the accessed video based on the any two adjacent frames and their respective reconstructed frame, wherein the reconstructed accessed video is associated with a second framerate higher than the threshold framerate.

15 . The computer-readable medium of claim 14 , wherein the accessing the video captured by the one or more cameras comprises: receiving, via a wireless network, the accessed video transmitted from the one or more cameras.

16 . The computer-readable medium of claim 15 , wherein the wireless network is associated with a bandwidth lower than a threshold bandwidth.

17 . The computer-readable medium of claim 14 , wherein the one or more cameras are associated with a wearable device.

18 . The computer-readable medium of claim 14 , wherein:
the instructions, when executed, further cause: extracting respective features for each frame of the any two adjacent frames, and the respective alignments are determined based on the respective extracted features.

19 . A system comprising:
one or more processors; and a non-transitory memory coupled to the one or more processors comprising instructions executable by the one or more processors, the one or more processors operable when executing the instructions to:
access a video captured by one or more cameras, wherein the accessed video is associated with a first framerate lower than a threshold framerate;
for any two adjacent frames of the accessed video:
generate respective warped frames for each frame of the any two adjacent frames based on a respective optical flow associated with each frame of the any two adjacent frames;
fuse the respective warped frames to generate an intermediate frame;
determine respective alignments associated with each frame of the any two adjacent frames based on aligning each of the any two adjacent frames to the intermediate frame;
fuse the respective determined al

Assignees

• Meta Platforms Inc

Inventors

• Xiang Xiaoyu
• Cotoros Ingrid Anda
• Ranjan Rakesh
• Chandra Vikas

Classifications

• G06T7/33

  using feature-based methods · CPC title

• G06V10/806

  of extracted features · CPC title

• G06T2207/10016

  Video; Image sequence · CPC title

• H04N7/0127

  by changing the field or frame frequency of the incoming video signal, e.g. frame rate converter · CPC title

• G06T7/38 Primary

  Registration of image sequences · CPC title