Generating synthetic video frames using optical flow

Patent metadata

Field	Value
Publication number	US-9626770-B2
Application number	US-201514683646-A
Country	US
Kind code	B2
Filing date	Apr 10, 2015
Priority date	Apr 10, 2015
Publication date	Apr 18, 2017
Grant date	Apr 18, 2017

Abstract

A novel method of using optical flow algorithm that maximizes the benefit of optical flow synthetic frames while minimizing the associated computation cost is provided. When using optical flow to produce synthetic frames between two actual/recorded frames, the method computes a set of estimates of optical flows (or a flow estimate) between the two frames. These flow estimates are then used to compute all synthetic frames that are needed between the two actual frames by interpolation, which creates each synthetic frame based on its temporal distances from the pair of actual frames.

First claim

What is claimed is:

1. A method of processing media content, the method comprising:
receiving first and second video frames of a media content, the first video frame associated with a first timing instant and the second video frame associated with a second timing instant;
receiving a retiming request identifying a first set of timing instants that a video frame is scheduled to be displayed during a playback of the media content;
computing a set of optical flow estimates between the first video frame and the second video frame, each optical flow estimate computed at a timing instant in a second set of timing instants between the first timing instant and the second timing instant; and
computing a set of synthetic frames between the first video frame and the second video frame, each synthetic frame interpolated at a timing instant identified by the first set of timing instants between the first and second timing instants by using the set of computed optical flow estimates.

2. The method of claim 1 , wherein computing a synthetic frame at a particular timing instant comprises computing an interpolated optical field that is interpolated from the set of computed optical flow estimates at the particular timing instant.

3. The method of claim 1 , wherein the first and second video frames are consecutive video frames in the media content.

4. The method of claim 1 , wherein a number of timing instances in the second set of timing instants is a fixed number selected based on availability of computing resources.

5. The method of claim 1 , wherein the timing request comprises a request for one of fast forward, slow motion, reverse, pause, and conforming frame rate during the playback of the media content.

6. The method of claim 1 , wherein each of the first set of timing instants is also a timing instant for which the media content does not have a corresponding actual frame.

7. The method of claim 1 , wherein the first set of timing instants is different from the second set of timing instants.

8. The method of claim 1 , wherein a number of computed optical flow fields is less than a number of synthetic frames between the first and second video frames.

9. The method of claim 1 , wherein the second set of timing instants is a subset of the first set of timing instants, wherein computing the set of optical flow estimates comprises:
computing a first set of optical flow estimates at each of the second set of timing instants using an optical flow estimating algorithm; and
interpolating, based on the computed first set of optical flow estimates, a second set of optical flow estimates at each of a third set of timing instants, the third set of timing instants comprising every timing instant that is in the first set of timing instant sand not in the second set of timing instants.

10. An apparatus comprising:
a set of processing units;
a non-transitory machine readable medium storing a program for processing media content, the program executable by at least one of the processing units, the program comprising sets of instructions for:
receiving first and second video frames of a media content, the first video frame associated with a first timing instant and the second video frame associated with a second timing instant;
receiving a retiming request identifying a first set of timing instants that a video frame is scheduled to be displayed during a playback of the media content;
computing a set of optical flow estimates between the first video frame and the second video frame, each optical flow estimate computed at a timing instant in a second set of timing instants between the first time-slot timing instant and the second playback slot timing instant; and
computing a set of synthetic frames between the first video frame and the second video frame, each synthetic frame interpolated at a timing instant identified by the first set of timing instants between the first and second timing instants by using the set of computed optical flow estimates; and

a display device for displaying the set of synthetic frames.

11. The apparatus of claim 10 , wherein the set of instructions for computing a synthetic frame at a particular timing instant comprises a set of instructions for computing an interpolated optical field that is interpolated from the set of computed optical flow estimates at the particular timing instant.

12. The apparatus of claim 10 , wherein the first and second video frames are consecutive video frames in the media content.

13. The apparatus of claim 10 , wherein a number of timing instances in the second set of timing instants is a fixed number selected based on availability of computing resources.

14. The apparatus of claim 10 , wherein the retiming request comprises a request for one of fast forward, slow motion, reverse, pause, and conforming frame rate during the playback of the media content.

15. The apparatus of claim 10 , wherein each of the first set of timing instants is an instant that the display device is scheduled to display a video frame but the media content does not have a corresponding actual frame.

16. The apparatus of claim 10 , wherein the first set of timing instants is different from the second set of timing instants.

17. The apparatus of claim 10 , wherein a number of computed optical fields is less than a number of synthetic frames between the first and second video frames.

18. The apparatus of claim 10 , wherein the second set of timing instants is a subset of the first set of timing instants, wherein the set of instructions for computing the set of optical flow estimates comprises sets of instructions for:
computing a first set of optical flow estimates at each of the second set of timing instants using an optical flow estimating algorithm; and
interpolating, based on the computed first set of optical flow estimates, a second set of optical flow estimates at each of a third set of timing instants comprising every timing instant that is in the first set of timing instants and not in the second set of timing instants.

19. An apparatus comprising:
a set of processing units;
a non-transitory machine readable medium storing a program for processing media content, the program executable by at least one of the processing units, the program comprising:
a retiming engine for:
receiving a retiming request; and
specifying a first set of timing instants that a video frame is scheduled to be displayed during a playback of the media content;
an interpolation engine for:
computing a first set of optical flow estimates at each of a second set of timing instants using an optical flow estimating algorithm, the second set of timing instants a subset of the first set of timing instants; and
interpolating, based on the computed first set of optical flow estimates, a second set of optical flow estimates at each of a third set of timing instants, the third set of timing instants comprising every timing instant that is in the first set of timing instants and not in the second set of timing instants; and
generating a set synthetic frames based on the second interpolated set of optical flow estimates; and

a playback engine for playing back the synthetic frames.

20. The apparatus of claim 19 , wherein the retiming engine is part of a media editing application.

21. The apparatus of claim 19 , wherein a number of the first set of computed optical flow estimates is less than a number of the second set of interpolated optical flow estimates.

Assignees

• Apple Inc

Inventors

• Taggart Angus M
• Beale Eric S

Classifications

• H04N7/0137 Primary

  dependent on presence/absence of motion, e.g. of motion zones (H04N7/014 takes precedence; movement detection in television signals H04N5/144) · CPC title

• G06T7/207

  for motion estimation over a hierarchy of resolutions (multi-resolution motion estimation or hierarchical motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/53) · CPC title

• G06K9/36

  Physics · mapped topic

• G06K9/00711

  Physics · mapped topic

• H04N5/144

  Movement detection (for video coding H04N19/503; analysis of motion in general G06T7/20) · CPC title