Apparatus and methods for the selection of one or more frame interpolation techniques

Patent metadata

Field	Value
Publication number	US-10057538-B2
Application number	US-201715407089-A
Country	US
Kind code	B2
Filing date	Jan 16, 2017
Priority date	Jan 16, 2017
Publication date	Aug 21, 2018
Grant date	Aug 21, 2018

Abstract

Methods and apparatus for the generation of interpolated frames of video data. In one embodiment, the interpolated frames of video data are generated by obtaining two or more frames of video data; performing Lagrangian interpolation on one or more portions of the obtained two or more frames of video data to generate a Lagrangian interpolated image; performing Eulerian interpolation on one or more portions of the obtained two or more frames to generate a Eulerian interpolated image; and when the Lagrangian interpolated image and the Eulerian interpolated image should be combined, computing an average interpolated image using the Lagrangian interpolated image and the Eulerian interpolated image; otherwise, selecting either the Lagrangian interpolated image or the Eulerian interpolated image; and generating an interpolated frame of video data using one or more of the average interpolated image, the Lagrangian interpolated image, or the Eulerian interpolated image.

First claim

What is claimed:

1. A computerized apparatus for generation of interpolated frames of video data, the computerized apparatus comprising:
a processing apparatus; and
a storage apparatus in data communication with the processing apparatus, the storage apparatus having a non-transitory computer readable medium comprising instructions which are configured to, when executed by the processing apparatus, cause the computerized apparatus to:
obtain two or more frames of video data;
perform Lagrangian interpolation on one or more portions of the obtained two or more frames of video data to generate a Lagrangian interpolated image;
perform Eulerian interpolation on the one or more portions of the obtained two or more frames to generate a Eulerian interpolated image; and
when the Lagrangian interpolated image and the Eulerian interpolated image should be combined, compute an average interpolated image using the Lagrangian interpolated image and the Eulerian interpolated image;
otherwise, select either the Lagrangian interpolated image or the Eulerian interpolated image; and
generate an interpolated frame of video data using one or more of the average interpolated image, the Lagrangian interpolated image, or the Eulerian interpolated image.

2. The computerized apparatus of claim 1 , wherein the non-transitory computer readable medium further comprises instructions which are configured to, when executed by the processing apparatus, cause the computerized apparatus to:
generate the interpolated frame of video data using two or more of the average interpolated image, the Lagrangian interpolated image, or the Eulerian interpolated image.

3. The computerized apparatus of claim 2 , wherein the non-transitory computer readable medium further comprises instructions which are configured to, when executed by the processing apparatus, cause the computerized apparatus to:
generate the interpolated frame of video data using the average interpolated image, the Lagrangian interpolated image, and the Eulerian interpolated image.

4. The computerized apparatus of claim 1 , wherein the one or more portions of the obtained two or more frames are selected on a per-pixel basis.

5. The computerized apparatus of claim 1 , wherein the one or more portions of the obtained two or more frames are selected on a per-region basis, at least one of the one or more portions of the obtained two or more frames comprising a super pixel region.

6. The computerized apparatus of claim 1 , wherein at least one of the obtained two or more frames of video data comprises a previously generated interpolated frame of video data.

7. The computerized apparatus of claim 1 , wherein the non-transitory computer readable medium further comprises instructions which are configured to, when executed by the processing apparatus, cause the computerized apparatus to:
compute a criteria between a first frame of the two or more frames of video data and a second frame of the two or more frames of video data;
when the criteria is met, determine that the Lagrangian interpolated image and the Eulerian interpolated image should be combined; and
when the criteria is not met, determine that either the Lagrangian interpolated image or the Eulerian interpolated image should be chosen.

8. The computerized apparatus of claim 7 , wherein the criteria comprises a distance measure between the first frame of the two or more frames of video data and the second frame of the two or more frames of video data.

9. The computerized apparatus of claim 7 , wherein the criteria comprises a color similarity measure between the first frame of the two or more frames of video data and the second frame of the two or more frames of video data.

10. The computerized apparatus of claim 7 , wherein the criteria comprises an entropy measure between the first frame of the two or more frames of video data and the second frame of the two or more frames of video data.

11. A method for generation of interpolated frames of video data, the method comprising:
obtaining two or more frames of video data;
performing Lagrangian interpolation on one or more portions of the obtained two or more frames of video data to generate a Lagrangian interpolated image;
performing Eulerian interpolation on the one or more portions of the obtained two or more frames to generate a Eulerian interpolated image; and
when the Lagrangian interpolated image and the Eulerian interpolated image should be combined, computing an average interpolated image using the Lagrangian interpolated image and the Eulerian interpolated image;
otherwise, selecting either the Lagrangian interpolated image or the Eulerian interpolated image; and
generating an interpolated frame of video data using one or more of the average interpolated image, the Lagrangian interpolated image, or the Eulerian interpolated image.

12. The method of claim 11 , further comprising:
computing a criterion between a first frame of the two or more frames of video data and a second frame of the two or more frames of video data;
minimizing a cost function associated with the computed criterion;
weighting a first one of the one or more interpolation methodologies more than a second one of the one or more interpolation methodologies; and
combining the at least two differing interpolation methodologies.

13. The method of claim 11 , wherein the obtaining of the two or more frames comprises selecting the two or more frames on (i) a per-pixel basis or (ii) a per-region basis.

14. The method of claim 11 , wherein the obtaining of the two or more frames comprises obtaining a previously generated interpolated frame of video data.

15. The method of claim 11 , further comprising:
computing a criteria between a first frame of the two or more frames of video data and a second frame of the two or more frames of video data;
when the criteria is met, determining that the Lagrangian interpolated image and the Eulerian interpolated image should be combined; and
when the criteria is not met, determining that either the Lagrangian interpolated image or the Eulerian interpolated image should be chosen.

16. A non-transitory computer-readable apparatus comprising a storage medium, the storage medium having a computer program thereon, the computer program comprising a plurality of instructions configured to, when executed by a processor apparatus, cause the processor apparatus to:
obtain two or more frames of video data;
perform Lagrangian interpolation on one or more portions of the obtained two or more frames of video data to generate a Lagrangian interpolated image;
perform Eulerian interpolation on the one or more portions of the obtained two or more frames to generate a Eulerian interpolated image; and
responsive to a determination that the Lagrangian interpolated image and the Eulerian interpolated image should be combined, compute an average interpolated image using the Lagrangian interpolated image and the Eulerian interpolated image;
responsive to a determination that the Lagrangian interpolated image and the Eulerian interpolated image should not be combined, select either the Lagrangian interpolated image or the Eulerian interpolated image; and
generate an interpolated frame of video data using one or more of the average interpolated image, the Lagrangian interpolated image, or the Eulerian interpolated image.

17. The non-transitory computer-readable apparatus of claim 16 , wherein the determination that the Lagrangian interpolated image and the Eulerian interpolated image should be combined is based on one or more of (i) a distance

Assignees

• Gopro Inc

Inventors

• Staranowicz Aaron
• Lustig Ryan

Classifications

• H04N11/20

  Conversion of the manner in which the individual colour picture signal components are combined, e.g. conversion of colour television standards · CPC title

• H04N7/014 Primary

  involving the use of motion vectors (motion estimation and compensation in video coding H04N19/51) · CPC title