Method for camera motion estimation and correction

Patent metadata

Field	Value
Publication number	US-9967463-B2
Application number	US-201414906869-A
Country	US
Kind code	B2
Filing date	Jul 24, 2014
Priority date	Jul 24, 2013
Publication date	May 8, 2018
Grant date	May 8, 2018

Abstract

A motion estimation and correction system and methods are shown comprising: an image acquisition device configured to acquire an image via scanning an image frame over a period of time, an inertial measurement unit configured to measure at least one of a position, an orientation, and a movement of the image acquisition device during the period of time and output an indication of the movement as detected; and a state estimation module, operatively coupled to the inertial measurement unit and the image acquisition device, configured to estimate a state related to at least one of position and orientation of the image acquisition device based on the at least one of the position, the orientation, and the movement. In one example, systems and methods include using an inertial measurement unit along with an image acquisition device. In one example, the image acquisition device includes a rolling-shutter camera.

First claim

What is claimed is:

1. A system, comprising:
an image acquisition device configured to acquire an image via scanning an image frame over a period of time;
an inertial measurement unit configured to measure at least one of a position, an orientation, and a movement of the image acquisition device during the period of time and output an indication of the movement as detected; and
a state estimation module, operatively coupled to the inertial measurement unit and the image acquisition device, configured to estimate a state related to at least one of position and orientation of the image acquisition device based on the at least one of the position, the orientation, and the movement, wherein the state estimation module is configured to use a parametric model of the errors in the state estimates to effect the computation of the state updates, and to vary a number of terms in the parametric model of the errors in response to a magnitude of one or more errors.

2. The system of claim 1 , wherein the state estimation module is configured to output the state as estimated, and further comprising a user interface configured to display the state as output from the state estimation module.

3. The system of claim 1 , wherein the inertial measurement unit is rigidly connected to the image acquisition device.

4. The system of claim 1 , wherein the image acquisition device is a rolling-shutter camera.

5. The system of claim 1 , wherein the state estimation module comprises a processor configured to apply a parametric model to the movement to estimate the state.

6. The system of claim 1 , wherein the inertial measurement unit includes at least one of an accelerometer, a gyroscope, and a magnetometer.

7. The system of claim 1 , wherein the state estimation module is configured to output the state as estimated, and further comprising a state correction module operatively coupled to the state estimation module and configured to correct the image according to the state as output from the state estimation module.

8. The system of claim 1 , wherein the state estimation module is configured to use a parametric model developed on a basis of temporal functions.

9. A state estimation circuit, comprising:
a memory module configured to store a description of a movement of an image acquisition device as computed based on an output of an inertial measurement unit during a period of time; and
a processor configured to estimate a state related to at least one of position and orientation of the image acquisition device, wherein the processor is configured to apply a parametric model of one or more errors in a state estimate to effect a computation of a state update, and to vary a number of terms in the parametric model of the errors in response to a magnitude of one or more errors;
wherein the image acquisition device is configured to acquire the image via scanning an image frame over the period of time.

10. The state estimation circuit of claim 9 , wherein the state estimation circuit is configured to output the state as estimated to a user interface configured to display the state as output.

11. The state estimation circuit of claim 9 , wherein the image acquisition device is a rolling-shutter camera.

12. The state estimation circuit of claim 9 , wherein the inertial measurement unit includes at least one of an accelerometer, a gyroscope, and a magnetometer.

13. The state estimation circuit of claim 9 , further configured to correct the image according to the state.

14. The state estimation circuit of claim 9 , wherein the processor is configured to model motion of the image acquisition device during the period of time, in order to compute measurement residuals for updating a state estimate.

15. The state estimation circuit of claim 9 , wherein the processor is configured to apply a parametric model developed on a basis of temporal functions.

16. A method, comprising:
acquiring an image from an image acquisition device by scanning an image frame over a period of time;
using an inertial measurement unit to acquire at least one of a position, an orientation, and a movement of the image acquisition device during the period of time;
estimating a state relating to the image acquisition device with a state estimation module, wherein estimating a state includes using a parametric model of one or more errors in a state estimate to effect a computation of a state update; and
varying a number of error estimation terms used to estimate the state relating to the image acquisition device.

17. The method of claim 16 , further comprising displaying, on a user interface, the state as estimated from the state estimation module.

18. The method of claim 16 , wherein using the inertial measurement unit includes using an inertial measurement unit that is rigidly connected to the image acquisition device.

19. The method of claim 16 , wherein acquiring the image from an image acquisition device includes acquiring an image from a rolling-shutter camera.

20. The method of claim 16 , wherein estimating the state relating to the image acquisition device includes utilizing a processor configured to apply a parametric model to the movement to estimate the state.

21. The method of claim 16 , wherein using the inertial measurement unit includes using at least one of an accelerometer, a gyroscope, and a magnetometer.

22. The method of claim 16 , further comprising using an state correction module to correct the image according to the state as output from the state estimation module.

23. The method of claim 16 , wherein estimating a state includes using inertial measurements to model motion of the image acquisition device during the period of time to compute measurement residuals for updating a state estimate.

24. The method of claim 16 , wherein estimating a state includes using a parametric model developed on a basis of temporal functions.

Assignees

• Univ California

Inventors

• Mourikis Anastasios
• Li Mingyang

Classifications

• H04N23/689 Primary

  Motion occurring during a rolling shutter mode · CPC title

• H04N23/63

  by using electronic viewfinders · CPC title

• H04N23/683 Primary

  performed by a processor, e.g. controlling the readout of an image memory · CPC title

• H04N23/6812

  based on additional sensors, e.g. acceleration sensors · CPC title

• H04N5/23258

  Electricity · mapped topic