Combined mechanical and electronic image stabilization

Patent metadata

Field	Value
Publication number	US-12063439-B2
Application number	US-202217982275-A
Country	US
Kind code	B2
Filing date	Nov 7, 2022
Priority date	Dec 11, 2017
Publication date	Aug 13, 2024
Grant date	Aug 13, 2024

Abstract

Systems and methods are disclosed for image signal processing. For example, methods may include, based on a sequence of orientation estimates for an image sensor and an orientation setpoint, invoking a mechanical stabilization system to adjust an orientation of the image sensor toward the orientation setpoint; receiving an image from the image sensor; determining an orientation error between the orientation of the image sensor and the orientation setpoint during capture of the image; and, based on the orientation error, invoking an electronic image stabilization module to correct the image for a rotation corresponding to the orientation error to obtain a stabilized image.

First claim

What is claimed is:

1. A system comprising:
an image sensor;
one or more motion sensors configured to detect motion of the image sensor;
a mechanical stabilization system configured to control an orientation of the image sensor;
an electronic image stabilization module configured to correct images for motion of the image sensor; and
a processing apparatus that is configured to:
based on sensor data from the one or more motion sensors and an orientation setpoint for the image sensor, invoke the mechanical stabilization system to adjust the orientation of the image sensor;
receive an image from the image sensor;
store a sequence of images captured after the image in a buffer;
determine an orientation error between the orientation of the image sensor and the orientation setpoint during capture of the image;
determine a rotation corresponding to the orientation error based on a sequence of orientation estimates corresponding to the sequence of images, wherein the orientation estimates are based on sensor data from the one or more motion sensors; and
based on the orientation error, invoke the electronic image stabilization module to correct the image to obtain a stabilized image, in which the electronic image stabilization module corrects the image for the rotation corresponding to the orientation error.

2. The system of claim 1 , further comprising:
a drone that is coupled to a housing of the image sensor by the mechanical stabilization system.

3. The system of claim 1 , in which the processing apparatus is further configured to:
determine a trajectory based on the sequence of orientation estimates corresponding to the sequence of images; and
determine the rotation corresponding to the orientation error based on the trajectory.

4. The system of claim 3 , in which the processing apparatus is further configured to:
determine a sequence of orientation errors based on the sequence of orientation estimates corresponding to the sequence of images and the orientation setpoint; and
apply a filter to the sequence of orientation errors to obtain the trajectory.

5. The system of claim 1 , in which the image is captured with an electronic rolling shutter, the orientation error is a first orientation error associated with a first portion of the image, and the processing apparatus is further configured to:
determine a second orientation error between the orientation of the image sensor and the orientation setpoint during capture of a second portion of the image; and
based on the second orientation error, invoke the electronic image stabilization module to correct the second portion of the image for a rotation corresponding to the second orientation error to obtain the stabilized image.

6. The system of claim 1 , in which the one or more motion sensors include encoders configured to detect a position and an orientation of the image sensor relative to a movable platform.

7. The system of claim 1 , in which the orientation setpoint includes a quaternion.

8. The system of claim 1 , in which the electronic image stabilization module is implemented by software executed by the processing apparatus.

9. A method comprising:
based on a sequence of orientation estimates for an image sensor and an orientation setpoint, invoking a mechanical stabilization system to adjust an orientation of the image sensor toward the orientation setpoint;
receiving an image from the image sensor;
storing a sequence of images captured after the image in a buffer;
determining an orientation error between the orientation of the image sensor and the orientation setpoint during capture of the image;
determining a rotation corresponding to the orientation error based on orientation estimates from the sequence of orientation estimates corresponding to the sequence of images; and
based on the orientation error, invoking an electronic image stabilization module to correct the image to obtain a stabilized image, in which the electronic image stabilization module corrects the image for the rotation corresponding to the orientation error.

10. The method of claim 9 , further comprising:
determining a trajectory based on the sequence of orientation estimates corresponding to the sequence of images; and
determining the rotation corresponding to the orientation error based on the trajectory.

11. The method of claim 10 , further comprising:
determining a sequence of orientation errors based on the sequence of orientation estimates corresponding to the sequence of images and the orientation setpoint; and
applying a filter to the sequence of orientation errors to obtain the trajectory.

12. The method of claim 9 , in which the image is captured with an electronic rolling shutter, the orientation error is a first orientation error associated with a first portion of the image, and further comprising:
determining a second orientation error between the orientation of the image sensor and the orientation setpoint during capture of a second portion of the image; and
based on the second orientation error, invoking the electronic image stabilization module to correct the second portion of the image for a rotation corresponding to the second orientation error to obtain the stabilized image.

13. The method of claim 9 , in which an orientation in the sequence of orientation estimates includes an estimate of orientation of the image sensor with respect to a movable platform and an estimate of orientation of the image sensor with respect to gravity.

14. The method of claim 9 , in which the mechanical stabilization system includes gimbals and motors controlled by proportional integral derivative controllers.

15. A system comprising:
an image sensor configured to capture an image;
a mechanical stabilization system configured to control an orientation of the image sensor to match an orientation setpoint;
an electronic image stabilization module configured to correct the image for motion of the image sensor corresponding to orientation errors between the orientation of the image sensor and the orientation setpoint during capture of the image;
a buffer configured to store a sequence of images captured by the image sensor;
a fast trajectory generator configured to determine a sequence of rotations based on orientation errors corresponding to images in the buffer, wherein the electronic image stabilization module is configured to use a rotation from the sequence of rotations corresponding to an oldest image stored in the buffer; and
a drone that is coupled to a housing of the image sensor by the mechanical stabilization system.

16. The system of claim 15 , in which the mechanical stabilization system includes gimbals and motors controlled by proportional integral derivative controllers.

17. The system of claim 15 , further comprising:
a motion tracking module including one or more motion sensors configured to detect motion of the image sensor and determine a sequence of orientation estimates for the image sensor, wherein the mechanical stabilization system is configured to take the sequence of orientation estimates as input and use the sequence of orientation estimates as feedback for controlling the orientation of the image sensor.

18. The system of claim 17 , in which the fast trajectory generator is configured to:
determine a trajectory based on orientation estimates from the sequence of orientation estimates corresponding to the sequence of images; and
determine the rotation from the sequence of rotations corresponding to the oldest image stored in the buffer based on the trajector

Assignees

• Gopro Inc

Inventors

• Omari Sammy
• Gohl Pascal
• Enke Joseph A
• Moskovchenko Stepan
• Tankersley Benjamin P

Classifications

• H04N23/55

  Optical parts specially adapted for electronic image sensors; Mounting thereof · CPC title

• H04N23/6812

  based on additional sensors, e.g. acceleration sensors · CPC title

• H04N23/687

  by shifting the lens or sensor position · CPC title

• H04N23/685

  performed by mechanical compensation · CPC title

• H04N23/683

  performed by a processor, e.g. controlling the readout of an image memory · CPC title