Electronic image stabilization frequency estimator

Patent metadata

Field	Value
Publication number	US-9756249-B1
Application number	US-201615140391-A
Country	US
Kind code	B1
Filing date	Apr 27, 2016
Priority date	Apr 27, 2016
Publication date	Sep 5, 2017
Grant date	Sep 5, 2017

Abstract

Disclosed are a system and a method for determining enabling or disabling electronic image stabilization (EIS) for a video frame. An image sensor of a camera system captures a video stream that comprises a plurality of video frames. An image processor determines availability of a computational resource that may process application of EIS on each video frame. Simultaneously, the image processor receives motion data of the camera system from a gyroscope. Based on the computational resource availability, a motion frequency threshold is determined. Based on the gyroscope motion data, a motion frequency of each video frame is estimated. The estimated motion frequency is compared to the determined motion frequency threshold. If the estimated motion frequency is greater than the determined motion frequency threshold, application of EIS is disabled. If the estimated motion frequency is less than or equal to the determined motion frequency threshold, application of EIS is enabled.

First claim

What is claimed is:

1. A method for controlling electronic image stabilization, the method comprising:
capturing a video stream;
determining an availability of a computational resource for performing the electronic image stabilization on one or more frames of the video stream;
receiving data from a gyroscope, the gyroscope connected to an imaging device, wherein the data from the gyroscope includes sensed motion of the imaging device;
determining a baseline motion frequency threshold based on the availability of the computational resource;
estimating a motion frequency of the imaging device during capture of the video stream from the data received from the gyroscope;
comparing the estimated motion frequency of the imaging device to the determined baseline motion frequency threshold;
performing, in response to the estimated motion frequency meeting a first criteria based on the baseline motion frequency threshold, the electronic image stabilization on the one or more frames of the video stream; and
disabling, in response to the estimated motion frequency meeting a second criteria based on the baseline motion frequency threshold, performing the electronic image stabilization on the one or more frames of the video stream.

2. The method of claim 1 , wherein the gyroscope data further includes at least one of a positive displacement or a negative displacement, wherein the positive displacement indicates movement in a first direction and the negative displacement indicates movement in a second direction opposite the first direction.

3. The method of claim 2 , further comprising:
determining a number of polarity changes in the sensed motion from the data from the gyroscope during capture of the one or more frames, wherein the polarity changes indicate at least one of a change in motion from the first direction to the second direction or a change in motion from the second direction to the first direction.

4. The method of claim 1 , further comprising:
activating a switching element to enable or disable performing the electronic image stabilization on the one or more frames of the video stream.

5. The method of claim 1 , wherein estimating the motion frequency further comprises:
determining, for each video frame of the video stream, a number of polarity changes in the gyroscope data indicating at least one of a change from a motion in first direction that exceeds a first noise threshold to a motion in a second direction that exceeds a second noise threshold or a change from a motion in the second direction that exceeds the second noise threshold to a motion in the first direction that exceeds the first noise threshold;
determining a total number of polarity changes over a window of video frames of the video stream;
determining a total time of capture of the window of the video frames of the video stream; and
estimating the motion frequency based on the total number of polarity changes and the total time of the window of the video frames of the video stream.

6. The method of claim 1 , wherein the motion frequency is estimated on a sliding window of a plurality of video frames of the video stream.

7. The method of claim 1 , further comprising:
determining a lower motion frequency threshold below the baseline motion frequency threshold and an upper motion frequency threshold above the baseline motion frequency threshold; and
determining that the estimated motion frequency meets the first criteria if the estimated motion frequency is below the lower motion frequency threshold and determining that the estimated motion frequency meets the second criteria if the estimated motion frequency is above the upper motion frequency threshold.

8. The method of claim 7 , further comprising:
determining that the estimated motion frequency meets the first criteria if the estimated motion frequency is above the lower motion frequency threshold and below the upper motion frequency threshold and a last motion frequency met the first criteria; and
determining that the estimated motion frequency meets the second criteria if the estimated motion frequency is above the lower motion frequency threshold and below the upper motion frequency threshold and the last motion frequency met the second criteria.

9. The method of claim 1 , further comprising:
determining a tapering motion frequency threshold based on the baseline motion frequency threshold;
determining that the estimated motion frequency meets the first criteria if the estimated motion frequency is below the tapering motion frequency threshold, determining that the estimated motion frequency meets the second criteria if the estimated motion frequency is above the baseline motion frequency threshold, and determining that the estimated motion frequency meets a third criteria if the estimation motion frequency is between the tapering motion frequency threshold and the baseline motion frequency threshold; and
responsive to the estimated motion frequency meeting the third criteria, applying a partial electronic image stabilization to the one or more frames, a stabilization level of the partial electronic image stabilization varying inversely to the estimated motion frequency.

10. A non-transitory computer readable medium configured to store instructions for controlling electronic image stabilization, the instructions when executed by a processor cause the processor to:
capture a video stream;
determine an availability of a computational resource, the computational resource to perform the electronic image stabilization on one or more frames of the video stream;
receive data from a gyroscope, the gyroscope connected to an imaging device, wherein the data from the gyroscope includes sensed motion of the imaging device;
determine a baseline motion frequency threshold based on the availability of the computational resource;
estimate a motion frequency of the imaging device during capture of the video stream from the data received from the gyroscope;
compare the estimated motion frequency of the imaging device to the determined baseline motion frequency threshold;
perform, in response to the estimated motion frequency meeting a first criteria based on the baseline motion frequency threshold, the electronic image stabilization on the one or more frames of the video stream; and
disable, in response to the comparison that the estimated motion frequency meeting a second criteria based on the baseline motion frequency threshold, performing the electronic image stabilization on the one or more frames of the video stream.

11. The non-transitory computer readable storage medium of claim 10 , wherein the gyroscope data further includes at least one of a positive displacement or a negative displacement, wherein the positive displacement indicates movement in a first direction and the negative displacement indicates movement in a second direction opposite the first direction.

12. The non-transitory computer readable storage medium of claim 11 , wherein the instructions further cause the processor to determine a number of polarity changes in the sensed motion from the data from the gyroscope during capture of the one or more frames, wherein the polarity changes indicate at least one of a change in motion from a first direction to a second direction or a change in motion from the second direction to the first direction.

13. The non-transitory computer readable storage medium of claim 10 , wherein the instructions further cause the processor to activate a switching element to enable or disable performing the electronic image stabilization on the one or more frames of the video stream.

14. The non-transitory

Assignees

• Gopro Inc

Inventors

• Segapelli Loic

Classifications

• H04N5/144 Primary

  Movement detection (for video coding H04N19/503; analysis of motion in general G06T7/20) · CPC title

• H04N23/683

  performed by a processor, e.g. controlling the readout of an image memory · CPC title

• H04N23/698

  for achieving an enlarged field of view, e.g. panoramic image capture · CPC title

• H04N23/685

  performed by mechanical compensation · CPC title

• H04N23/6812 Primary

  based on additional sensors, e.g. acceleration sensors · CPC title