Real-time video stabilization for mobile devices based on on-board motion sensing

Patent metadata

Field	Value
Publication number	US-10027893-B2
Application number	US-201615151379-A
Country	US
Kind code	B2
Filing date	May 10, 2016
Priority date	May 10, 2016
Publication date	Jul 17, 2018
Grant date	Jul 17, 2018

Abstract

Real-time video stabilization for mobile devices based on on-board motion sensing. In accordance with a method embodiment of the present invention, a first image frame from a camera at a first time is accessed. A second image frame from the camera at a subsequent time is accessed. A crop polygon around scene content common to the first image frame and the second image frame is identified. Movement information describing movement of the camera in an interval between the first time and the second time is accessed. The crop polygon is warped to remove motion distortions of the second image frame is warped using the movement information. The warping may include defining a virtual camera that remains static when the movement of the camera is below a movement threshold. The movement information may describe the movement of the camera at each scan line of the second image frame.

First claim

What is claimed is:

1. A computer implemented method comprising:
accessing a first image frame from a camera at a first time;
accessing a second image frame from said camera at a subsequent time;
identifying a crop polygon around scene content common to said first image frame and said second image frame;
accessing movement information describing movement of said camera in an interval between said first time and said second time;
using said movement information, warping said crop polygon to remove motion distortions of said second image frame,
wherein said warping comprises defining a virtual camera that remains static when said movement of said camera is below a movement threshold;
defining a padding space around said crop polygon to adjust movement of said virtual camera;
damping movement of said virtual camera when said crop polygon is within an inner region of said padding space; and
accelerating said movement of said virtual camera when said crop polygon extends outside of said inner region of said padding space.

2. The computer implemented method of claim 1 wherein said subsequent time is in the range of two to five frames after said first time.

3. The computer implemented method of claim 1 wherein said movement information describes said movement of said camera at each scan line of said second image frame.

4. The computer implemented method of claim 1 wherein said warping comprises integrating said movement information to describe an orientation of said camera with respect to a global reference frame.

5. The computer implemented method of claim 1 further comprising:
slicing said crop polygon into at least two regions; and
applying a different warping to at least two of said at least two regions of said crop polygon.

6. An electronic system comprising:
an image sensor;
a motion sensing system configured to sense motion of said image sensor;
a processor coupled to said image sensor and to said motion sensing system;
a memory coupled to said processor, wherein said electronic system is configured to:
access a first image frame from a camera at a first time;
access a second image frame from said camera at a subsequent time;
identify a crop polygon around scene content common to said first image frame and said second image frame;
access movement information describing movement of said camera in an interval between said first time and said second time; and
warp said crop polygon, using said movement information, to remove motion distortions of said second image frame,
wherein said electronic system is configured to warp said crop polygon by defining a virtual camera that remains static when said movement of said camera is below a movement threshold,

wherein said electronic system is further configured to define a padding space around said crop polygon to adjust movement of a virtual camera, and
wherein said padding space comprises an inner region, corresponding to damping adjustment of said virtual camera, and an outer region, corresponding to increasing a rate of adjustment of said virtual camera.

7. The electronic system of claim 6 wherein said movement information describes said movement of said camera at each scan line of said second image frame.

8. The electronic system of claim 6 wherein said electronic system is further configured to integrate said movement information to describe an orientation of said camera with respect to a global reference frame.

9. The electronic system of claim 6 wherein said electronic system is further configured to define a virtual camera that remains static when said movement of said camera is relatively small.

10. The electronic system of claim 6 wherein said electronic system is further configured to:
slice said crop polygon into a plurality of regions; and
apply a different warping to at least two of said plurality of regions of said crop polygon.

11. An article of manufacture including a non-transitory computer readable medium having instructions stored thereon that, responsive to execution by an electronic system, cause said electronic system to perform operations comprising:
accessing a first image frame from a camera at a first time;
accessing a second image frame from said camera at a subsequent time;
identifying a crop polygon around scene content common to said first image frame and said second image frame;
accessing movement information describing movement of said camera in an interval between said first time and said second time; and
using said movement information, warping said crop polygon to remove motion distortions of said second image frame,
wherein said warping comprises defining a virtual camera that remains static when said movement of said camera is below a movement threshold,
defining a padding space around said crop polygon to adjust movement of said virtual camera;
damping movement of said virtual camera when said crop polygon is within an inner region of said padding space; and
accelerating said movement of said virtual camera when said crop polygon extends outside of said inner region of said padding space.

12. The article of manufacture of claim 11 wherein said movement information describes said movement of said camera at each scan line of said second image frame.

13. The article of manufacture of claim 11 wherein said warping further comprises integrating said movement information to describe an orientation of said camera with respect to a global reference frame.

14. The article of manufacture of claim 13 wherein said warping further comprises defining a virtual camera that remains static when said movement of said camera is relatively small.

Assignees

• Nvidia Corp

Inventors

• Bell Steven
• Troccoli Alejandro
• Pulli Kari

Classifications

• H04N23/683

  performed by a processor, e.g. controlling the readout of an image memory · CPC title

• H04N23/689

  Motion occurring during a rolling shutter mode · CPC title

• H04N23/6812

  based on additional sensors, e.g. acceleration sensors · CPC title

• H04N23/6842 Primary

  by controlling the scanning position, e.g. windowing · CPC title

• H04N23/6811

  based on the image signal · CPC title