Joint video stabilization and rolling shutter correction on a generic platform

Patent metadata

Field	Value
Publication number	US-9460495-B2
Application number	US-201213440993-A
Country	US
Kind code	B2
Filing date	Apr 6, 2012
Priority date	Apr 6, 2012
Publication date	Oct 4, 2016
Grant date	Oct 4, 2016

Abstract

In one embodiment, a video processing system 300 may filter a video data set to correct skew and wobble using a central processing unit 220 and a graphical processing unit 230 . The video processing system 300 may apply a rolling shutter effect correction filter to an initial version of a video data set. The video processing system 300 may simultaneously apply a video stabilization filter to the initial version to produce a final version video data set.

First claim

We claim:

1. A machine-implemented method, comprising:
determining a filtering apportionment distributing a first filtering task to a graphical processing unit and a second filtering task to a central processing unit based on a prior filter performance;
applying a rolling shutter effect correction filter to an initial version of a video data set;
warping an image of the initial version by applying a motion vector set using the graphical processing unit;
adjusting a warping constant on the motion vector set based on a previous iteration; and
applying a video stabilization filter to the initial version to produce a final version of the video data set.

2. The method of claim 1 , further comprising:
executing a motion estimation on the initial version using the graphical processing unit to create a down sample set.

3. The method of claim 1 , further comprising:
processing a down sample set of the initial version using the central processor unit to create a motion vector set.

4. The method of claim 1 , further comprising:
creating a preview proxy of the video data set using the rolling shutter effect correction filter and the video stabilization filter.

5. The method of claim 1 , further comprising:
caching a preview proxy set of the video data set.

6. The method of claim 1 , further comprising:
receiving a user selection of a preview proxy of a preview proxy set; and
creating the final version based on the user selection.

7. The method of claim 1 , further comprising:
setting a filter parameter for the rolling shutter effect correction filter.

8. The method of claim 1 , further comprising:
tracking a load state of the video data set using a load status state machine.

9. A tangible machine-readable medium having a set of instructions detailing a method stored thereon that when executed by one or more processors cause the one or more processors to perform the method, the method comprising:
determining a filtering apportionment distributing a first filtering task to a graphical processing unit and a second filtering task to a central processing unit based on a prior filter performance;
applying a rolling shutter effect correction filter to an initial version of a video data set;
warping an image of the initial version by applying a motion vector set using the graphical processing unit;
adjusting a warping constant on the motion vector set based on a previous iteration; and
applying a video stabilization filter to the initial version to produce a final version of the video data set.

10. The tangible machine-readable medium of claim 9 , wherein the method further comprises:
executing a motion estimation on the initial version using the graphical processing unit to create a down sample set.

11. The tangible machine-readable medium of claim 9 , wherein the method further comprises:
processing a down sample set of the initial version using the central processor unit to create a motion vector set.

12. The tangible machine-readable medium of claim 9 , wherein the method further comprises:
creating a preview proxy of the video data set using the rolling shutter effect correction filter and the video stabilization filter; and
caching the preview proxy set of the video data set.

13. The tangible machine-readable medium of claim 9 , wherein the method further comprises:
receiving a user selection of a preview proxy of a preview proxy set; and
creating the final version based on the user selection.

14. The tangible machine-readable medium of claim 9 , wherein the method further comprises:
setting a filter parameter for the rolling shutter effect correction filter.

15. The tangible machine-readable medium of claim 9 , wherein the method further comprises:
tracking a load state of the video data set using a load status state machine.

16. A video processing system, comprising:
a central processing unit configured to determine a filtering apportionment distributing a first filtering task to a graphical processing unit and a second filtering task to the central processing unit based on a prior filter performance and to apply a rolling shutter effect correction filter to an initial version of a video data set; and
the graphical processing unit configured to warp an image of the initial version by applying a motion vector set, adjust a warping constant on the motion vector set based on a previous iteration, and apply a video stabilization filter to the initial version to produce a final version of the video data set.

17. The video processing system of claim 16 , further comprising:
a cache configured to store a preview proxy set of the video data set.

18. The video processing system of claim 16 , wherein the graphical processing unit is further configured to execute a motion estimation on the initial version to create a down sample set.

19. The video processing system of claim 16 , wherein the central processing unit is further configured to process a down sample set of the initial version to create the motion vector set having at least one row motion vector.

20. The video processing system of claim 19 , wherein the central processing unit is further configured to create at least one global motion transform from the at least one row motion vector.

Assignees

• Wu Yongjun
• Wozniak Matthew
• Baker Simon
• Negrila Catalin Alexandru
• Lanka Venkata S K Kamal
• Chin Kevin
• Kohlwey Brian
• Microsoft Technology Licensing Llc

Inventors

• Wu Yongjun
• Wozniak Matthew
• Baker Simon
• Negrila Catalin Alexandru
• Lanka Venkata S K Kamal
• Chin Kevin
• Kohlwey Brian

Classifications

• H04N23/683

  performed by a processor, e.g. controlling the readout of an image memory · CPC title

• H04N23/689

  Motion occurring during a rolling shutter mode · CPC title

• H04N23/6811

  based on the image signal · CPC title

• G06T5/10 Primary

  using non-spatial domain filtering · CPC title

• G06T2207/20016

  Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform · CPC title