Reduced latency video stabilization

Patent metadata

Field	Value
Publication number	US-9824426-B2
Application number	US-201113195776-A
Country	US
Kind code	B2
Filing date	Aug 1, 2011
Priority date	Aug 1, 2011
Publication date	Nov 21, 2017
Grant date	Nov 21, 2017

Abstract

Reduced latency video stabilization methods and tools generate truncated filters for use in the temporal smoothing of global motion transforms representing jittery motion in captured video. The truncated filters comprise future and past tap counts that can be different from each other and are typically less than those of a baseline filter providing a baseline of video stabilization quality. The truncated filter future tap count can be determined experimentally by comparing a smoothed global motion transform set generated by applying a baseline filter to a video segment to those generated by multiple test filter with varying future tap counts, then settings the truncated filter future tap count based on an inflection point on an error-future tap count curve. A similar approach can be used to determine the truncated filter past tap count.

First claim

We claim:

1. A computer-implemented method of digital video stabilization, the method comprising:
for at least one frame of a plurality of frames of a video segment, determining at least one global motion transform representing jittery video motion;
applying temporal smoothing to the at least one global motion transform to produce at least one smoothed global motion transform, the applying temporal smoothing utilizing a filter comprising a number of future taps and a number of past taps, the number of future taps and the number of past taps each being greater than zero, and the number of future taps being less than the number of past taps; and
processing the at least one of the frames of the video segment based on the at least one smoothed global motion transform to compensate for jittery video motion;
after the processing based on the at least one smoothed global motion transform, outputting the processed at least one of the frames of the video segment for display or subsequent processing.

2. The method of claim 1 , further comprising:
as part of the determining the at least one global motion transform, processing local motion vectors of each frame of the at least one frame of the plurality of frames of the video segment to produce the at least one global motion transform.

3. The method of claim 1 , wherein the at least one global motion transform is based on a similarity model of video motion.

4. The method of claim 1 , wherein the filter is a truncated smooth filter.

5. The method of claim 1 , wherein the number of future taps is greater than or equal to 2 and less than or equal to 4.

6. The method of claim 1 , wherein the number of future taps is determined from test result data indicating a tradeoff between stabilization quality and the number of future taps or past taps in the filter.

7. A computer-implemented method of determining a truncated filter future tap count for use in temporally smoothing global motion transforms representing jittery motion in a video segment, the method comprising:
determining global motion transforms for a plurality of frames in the video segment;
filtering the global motion transforms with a baseline filter comprising n future taps to generate a set of baseline smoothed global motion transforms;
filtering the global motion transforms with a plurality of test filters to generate a plurality of test smoothed global motion transform sets, each of the plurality of test filters having any of 0 through m future taps, wherein 0<m<n; and
determining a truncated filter future tap count based at least in part on the set of baseline smoothed global motion transforms and the plurality of test smoothed global motion transform sets, wherein the truncated filter future tap count is greater than zero, wherein the determining the truncated filter future tap count includes:
determining errors between the set of baseline smoothed global motion transforms and each of the plurality of test smoothed global motions transforms sets;
determining an inflection point along an error-future tap count curve based on the errors and the number of future taps in the plurality of test filters; and
selecting as the truncated filter future tap count, a number of future taps near the future tap count coordinate of the inflection point.

8. The method of claim 7 , wherein the determining global motion transforms, the filtering the global motion transforms with a baseline filter, and the filtering the global motion transforms with a plurality of test filters are performed for a plurality of additional video segments to generate additional errors, the error-future tap count curve based on an average of the errors and the additional errors.

9. The method of claim 7 , further comprising increasing the truncated filter future tap count in response to determining that a video processing pipeline can tolerate an increase in latency associated with the increase in the truncated filter future tap count.

10. Nonvolatile computer-readable storage media having encoded thereon computer-executable instructions for causing a computer system programmed thereby to perform digital video stabilization, the digital video stabilization comprising:
for at least one frame of a plurality of frames of a video segment, determining at least one global motion transform representing jittery video motion;
applying temporal smoothing to the at least one global motion transform to produce at least one smoothed global motion transform, the applying temporal smoothing utilizing a filter comprising a number of future taps and a number of past taps, the number of future taps and the number of past taps each being greater than zero, and the number of future taps being less than the number of past taps;
processing the at least one of the frames of the video segment based on the at least one smoothed global motion transform to compensate for jittery video motion; and
after the processing based on the at least one smoothed global motion transform, outputting the processed at least one of the frames of the video segment for display or subsequent processing.

11. The computer-readable storage media of claim 10 , wherein the digital video stabilization further comprises:
as part of the determining the at least one global motion transform, processing local motion vectors of each frame of the at least one frame of the plurality of frames of the video segment to produce the at least one global motion transform.

12. The computer-readable storage media of claim 10 , wherein the at least one global motion transform is based on a similarity model of video motion.

13. The computer-readable storage media of claim 10 , wherein the filter is a truncated smooth filter.

14. The computer-readable storage media of claim 10 , wherein the number of future taps is greater than or equal to 2 and less than or equal to 4.

15. The computer-readable storage media of claim 10 , wherein the number of future taps is determined from test result data indicating a tradeoff between stabilization quality and the number of future taps or past taps in the filter.

16. The computer-readable storage media of claim 10 , wherein the video segment is a first video segment having a first frame rate, and wherein the digital video stabilization further comprises:
determining that a second video segment has a second frame rate that is different from the first frame rate; and
scaling the number of future taps based on the second frame rate to produce a scaled number of future taps.

17. The computer-readable storage media of claim 10 , wherein the digital video stabilization further comprises:
increasing the number of future taps in response to determining that a video processing pipeline can tolerate an increase in latency associated with the increase in the number of future taps.

18. The method of claim 1 , wherein the video segment is a first video segment having a first frame rate, the method further comprising:
determining that a second video segment has a second frame rate that is different from the first frame rate; and
scaling the number of future taps based on the second frame rate to produce a scaled number of future taps.

19. The method of claim 1 , further comprising:
increasing the number of future taps in response to determining that a video processing pipeline can tolerate an increase in latency associated with the increase in the number of future taps.

20. A system for stabilizing digital video comprising:
a filter with a number of future taps and a number of past

Assignees

• Wu Yongjun
• Wozniak Matthew
• Sadhwani Shyam
• Thumpudi Naveen
• Microsoft Technology Licensing Llc

Inventors

• Wu Yongjun
• Wozniak Matthew
• Sadhwani Shyam
• Thumpudi Naveen

Classifications

• H04N23/683

  performed by a processor, e.g. controlling the readout of an image memory · CPC title

• H04N23/6811

  based on the image signal · CPC title

• G06T2207/20182

  Noise reduction or smoothing in the temporal domain; Spatio-temporal filtering · CPC title

• G06T5/002 Primary

  Physics · mapped topic

• H04N5/23267

  Electricity · mapped topic