Apparatus and methods for video image post-processing for segmentation-based interpolation

Patent metadata

Field	Value
Publication number	US-10134114-B2
Application number	US-201615270971-A
Country	US
Kind code	B2
Filing date	Sep 20, 2016
Priority date	Sep 20, 2016
Publication date	Nov 20, 2018
Grant date	Nov 20, 2018

Abstract

Apparatus and methods for video image post-processing for segmentation based interpolation. In one embodiment, a computerized apparatus is utilized in order to obtain a first frame of video data; segment one or more objects within the first frame of video data; obtain a second frame of video data; segment one or more objects within the second frame of video data; match at least a portion of the one or more objects within the first frame of video data with the one or more objects within the second frame of video data; compute the motion of the pixels for the matched portion of the one or more objects; compute the motion of pixels associated with a background image; and generate an interpolated frame of video data, the interpolated frame of video data residing temporally between the first frame of video data and the second frame of video data.

First claim

What is claimed:

1. A computerized apparatus configured to generate interpolated frames of video data, the apparatus comprising:
a video data interface configured to receive a plurality of frames of video data;
a processing apparatus in data communication with the video data interface; and
a storage apparatus in data communication with the processing apparatus, the storage apparatus having a non-transitory computer readable medium comprising instructions which are configured to, when executed by the processing apparatus, cause the computerized apparatus to:
obtain a first frame of video data via the video data interface;
segment one or more objects within the first frame of video data;
obtain a second frame of video data via the video data interface;
segment one or more objects within the second frame of video data;
match at least a portion of the one or more objects within the first frame of video data with the one or more objects within the second frame of video data;
compute a motion of pixels for the matched portion of the one or more objects;
compute a motion of pixels associated with a background image, the computed motion of pixels associated with the background image having the one or more objects segmented out prior to the computation of the motion of the pixels associated with the background image; and
generate an interpolated frame of video data via use of the computed motion of pixels for the matched portion of the one or more objects and the computed motion of pixels associated with the background image, the interpolated frame of video data residing temporally between the first frame of video data and the second frame of video data.

2. The computerized apparatus of claim 1 , wherein the computation of the motion of the pixels for the matched portion of the one or more objects comprises an independent computation of an optical flow field for each of the matched one or more objects.

3. The computerized apparatus of claim 2 , wherein the computation of the motion of the pixels associated with the background image comprises a computation of an optical flow field for the background image.

4. The computerized apparatus of claim 1 , wherein the match of the at least a portion of the one or more objects within the first frame of video data with the one or more objects within the second frame of video data comprises the use of a histogram-based method for determination of the matched one or more objects.

5. The computerized apparatus of claim 1 , wherein the generation of the interpolated frame of video data comprises generating a plurality of frames of interpolated video frame data, each of the plurality of frames of interpolated video frame data residing temporally between the first frame of video data and the second frame of video data.

6. The computerized apparatus of claim 5 , wherein the plurality of frames of interpolated video frame data are generated at even intervals between the first frame of video data and the second frame of video data.

7. The computerized apparatus of claim 5 , wherein the plurality of frames of interpolated video frame data are generated at irregular intervals between the first frame of video data and the second frame of video data.

8. A method of generating interpolated frames of video data, comprising:
causing obtainment of a pair of frames of video data;
causing segmentation of one or more objects within the pair of frames of video data, the segmentation resulting in one or more background images not having the one or more objects disposed therein;
causing calculation of a pixel motion for the one or more objects within the pair of frames of video data;
causing calculation of a pixel motion for the one or more background images not having the one or more objects disposed therein; and
causing generation of an interpolated frame of video data using the calculation of the pixel motion for the one or more objects within the pair of frames of video data and the calculation of the pixel motion for the one or more background images.

9. The method of claim 8 , wherein the causing of the segmentation of the one or more objects within the pair of frames of video data comprises causing the use of a pixel clustering methodology.

10. The method of claim 8 , wherein the causing of the segmentation of the one or more objects within the pair of frames of video data comprises causing the use of an edge detection methodology.

11. The method of claim 8 , wherein the causing of the segmentation of the one or more objects within the pair of frames of video data comprises causing the use of a variational methodology.

12. The method of claim 8 , wherein the causing of the segmentation of the one or more objects within the pair of frames of video data comprises causing the use of a convolutional neural network methodology.

13. The method of claim 8 , wherein the causing the segmentation of the one or more objects within the pair of frames of video data results in the generation of a plurality of objects within the pair of frames of video data, and wherein the method further comprises:
causing the performance of a pixel motion calculation on each of the plurality of objects in an independent manner.

14. A computing device, comprising:
logic configured to:
obtain a pair of frames of video data;
segment one or more objects within the pair of frames of video data, the segmentation of the one or more objects within the pair of frames of video data resulting in generation of a background image not having the one or more objects disposed therein;
match at least a portion of the one or more objects within a first of the pair of frames of video data with at least a portion of the one or more objects within a second of the pair of frames of video data;
compute a motion of pixels for the matched portion of the one or more objects;
compute a motion of pixels associated with the background image between the first of the pair of frames of video data and the second of the pair of frames of video data the background image having the one or more objects segmented out prior to the computation of the motion of the pixels associated with the background image; and
generate an interpolated frame of video data via use of the computed motion of the pixels for the matched portion of the one or more objects and use of the computed motion of the pixels associated with the background image, the interpolated frame of video data residing temporally between the first of the pair of frames of video data and the second of the pair of frames of video data.

15. The computing device of claim 14 , further comprising an image sensor, the image sensor configured to capture the pair of frames of video data.

16. The computing device of claim 14 , further comprising an interface module, the interface module comprising a display that is configured to display the interpolated frame of video data.

17. The computing device of claim 16 , wherein:
the interface module comprises a user interface, the user interface configured to receive a plurality of commands from a user in order to perform the motion of pixels computations for the matched portion of the one or more objects as well as the background image.

18. The computing device of claim 14 , wherein the logic comprises a processing unit and a non-transitory computer-readable storage medium having computer-executable program instructions stored thereon.

19. The computing device of claim 18 , wherein the match the at least the portion of the one or more objects within the first of the pair o

Assignees

• Gopro Inc

Inventors

• Adsumilli Balineedu Chowdary
• Lustig Ryan
• Staranowicz Aaron

Classifications

• G06T5/002 Primary

  Physics · mapped topic

• G06T3/4007

  based on interpolation, e.g. bilinear interpolation (image demosaicing G06T3/4015; edge-driven or edge-based scaling G06T3/403) · CPC title

• G06T5/003

  Physics · mapped topic

• G06T7/248 Primary

  involving reference images or patches · CPC title

• G06T2207/10024

  Color image · CPC title