Apparatus and methods for artifact detection and removal using frame interpolation techniques

Patent metadata

Field	Value
Publication number	US-10489897-B2
Application number	US-201715583794-A
Country	US
Kind code	B2
Filing date	May 1, 2017
Priority date	May 1, 2017
Publication date	Nov 26, 2019
Grant date	Nov 26, 2019

Abstract

Methods and apparatus for the generation of interpolated frames of video data. In one embodiment, the interpolated frames of video data are generated by obtaining two or more frames of video data from a video sequence; determining frame errors for the obtained two or more frames from the video sequence, determining whether the frame errors exceed a threshold value; performing a multi-pass operation; performing a single-pass operation; performing frame blending; performing edge correction; and generating the interpolated frame of image data.

First claim

What is claimed:

1. A method for the generation of interpolated frames, the method comprising:
obtaining a pair of frames from a video sequence, each of the frames of the pair having one or more errors associated therewith;
determining whether a magnitude of each of the errors exceeds a first threshold value, and based on a determination that the magnitude of each exceeds the first threshold value, performing at least a multi-pass operation;
responsive to the performing of the multi-pass operation, performing frame blending on the obtained pair of frames; and
generating an interpolated frame subsequent to the performance of the frame blending.

2. The method of claim 1 , wherein the one or more errors for the obtained pair of frames are determined by at least:
calculating a forward and a backward optical flow for the obtained pair of frames;
calculating a pair of estimated frames based on the calculated forward and backward optical flows; and
calculating the one or more errors for the obtained pair of frames from the video sequence based at least in part on the calculated pair of estimated frames.

3. The method of claim 2 , wherein the performance of the multi-pass operation further comprises:
dividing the calculated one or more errors into a plurality of tiles;
calculating statistical outliers on a per-tile basis based at least in part on the dividing; and
determining whether a number of the calculated statistical outliers exceeds a second threshold value on the per-tile basis.

4. The method of claim 3 , wherein the performance of the multi-pass operation further comprises:
calculating an updated optical flow for a given tile of the plurality of tiles;
performing a segmentation operation on at least a portion of the calculated statistical outliers; and
replacing an original optical flow calculation with the calculated updated optical flow for the given tile of the plurality of tiles.

5. The method of claim 4 , wherein the performance of the segmentation operation further comprises performing a nave segmentation operation.

6. The method of claim 4 , wherein the performance of the segmentation operation further comprises performing a boundary likelihood cost operation.

7. The method of claim 2 , further comprising performance of a single-pass operation, where the single-pass operation further comprises:
generating two or more occlusion masks using the calculated forward and backward optical flows;
combining the generated two or more occlusion masks; and
performing edge correction on the generated interpolated frame based at least in part on the combination of the generated two or more occlusion masks.

8. A non-transitory computer readable apparatus comprising a storage medium, the storage medium comprising a plurality of computer-executable instructions, the plurality of computer-executable instructions configured to, when executed by a processing apparatus:
obtain a pair of frames from a video sequence, each of the frames of the pair having one or more errors associated therewith;
determine one or more errors corresponding to the obtained pair of frames;
determine whether a magnitude of each of the errors exceeds a first threshold value;
perform at least a multi-pass operation based on a determination that the magnitude of each exceeds the first threshold value;
responsive to the performance of the multi-pass operation, perform frame blending on the obtained pair of frames from the video sequence; and
generate an interpolated frame subsequent to the performance of the frame blending.

9. The non-transitory computer readable apparatus of claim 8 , wherein the determination of the one or more errors further comprises:
calculation of a forward and a backward optical flow for the obtained pair of frames from the video sequence;
calculation of a pair of estimated frames based on the calculated forward and backward optical flows; and
calculation of the one or more errors corresponding to the obtained pair of frames from the video sequence based at least in part on the calculated pair of estimated frames.

10. The non-transitory computer readable apparatus of claim 9 , wherein the performance of the multi-pass operation further comprises:
division of each of the calculated one or more errors into a plurality of tiles;
calculation of statistical outliers on a per-tile basis based at least in part on the division; and
determination of whether a number of the calculated statistical outliers exceeds a threshold value on the per-tile basis.

11. The non-transitory computer readable apparatus of claim 10 , wherein the performance of the multi-pass operation further comprises:
calculation of an updated optical flow for a given tile of the plurality of tiles;
performance of a segmentation operation on at least a portion of the calculated statistical outliers; and
replace an original optical flow calculation with the calculated updated optical flow for the given tile of the plurality of tiles.

12. The non-transitory computer readable apparatus of claim 11 , wherein the performance of the segmentation operation further comprises performance of a naïve segmentation operation.

13. The non-transitory computer readable apparatus of claim 11 , wherein the performance of the segmentation operation further comprises performance of a boundary likelihood cost operation.

14. The non-transitory computer readable apparatus of claim 9 , wherein the performance of the single-pass operation further comprises:
generation of two or more occlusion masks using the calculated forward and backward optical flows;
combination of the generated two or more occlusion masks; and
performance of edge correction on the generated interpolated frame based at least in part on the combination of the generated two or more occlusion masks.

15. A computerized device configured to:
obtain two or more frames from a video sequence;
determine two or more errors corresponding to the obtained two or more frames from the video sequence;
based on a determination that a magnitude of each of the errors exceeds a first threshold value, perform a multi-pass operation and a single-pass operation on the two or more frames from the video sequence;
responsive to performance of the multi-pass operation, perform frame blending on the obtained two or more frames from the video sequence; and
responsive to the performance of the frame blending, generate an interpolated frame.

16. The computerized device of claim 15 , wherein the determining of the two or more errors for the obtained two or more frames from the video sequence comprises:
calculation of a forward and a backward optical flow for the obtained two or more frames from the video sequence;
calculation of estimated frames based on the calculated forward and backward optical flows; and
calculation of the two or more errors for the obtained two or more frames from the video sequence based on the calculated estimated frames.

17. The computerized device of claim 16 , wherein the performance of the single-pass operation that comprises:
generation of two or more occlusion masks based on the calculated forward and backward optical flows;
combination of the generated two or more occlusion masks; and
edge correction on the generated interpolated frame based at least in part on the combination of the generated two or more occlusion masks.

18. The computerized device of claim 15 , wherein the performance of the multi-pass operation comprises:
division of each of the two or more errors into a plurality of segments;

Assignees

• Gopro Inc

Inventors

• Staranowicz Aaron
• Lustig Ryan
• Chik Desmond

Classifications

• G06T5/20

  using local operators · CPC title

• H04N7/0137

  dependent on presence/absence of motion, e.g. of motion zones (H04N7/014 takes precedence; movement detection in television signals H04N5/144) · CPC title

• G06T2207/20192

  Edge enhancement; Edge preservation · CPC title

• G06T2207/10016

  Video; Image sequence · CPC title

• H04N7/014

  involving the use of motion vectors (motion estimation and compensation in video coding H04N19/51) · CPC title