Image stabilization using machine learning

Patent metadata

Field	Value
Publication number	US-2020382706-A1
Application number	US-202016995546-A
Country	US
Kind code	A1
Filing date	Aug 17, 2020
Priority date	Aug 31, 2018
Publication date	Dec 3, 2020
Grant date	—

Abstract

Techniques and systems are provided for machine-learning based image stabilization. In some examples, a system obtains a sequence of frames captured by an image capture device during a period of time, and collects motion sensor measurements calculated by a motion sensor associated with the image capture device based on movement of the image capture device during the period of time. The system generates, using a deep learning network and the motion sensor measurements, parameters for counteracting motions in one or more frames in the sequence of frames, the motions resulting from the movement of the image capture device during the period of time. The system then adjusts the one or more frames in the sequence of frames according to the parameters to generate one or more adjusted frames having a reduction in at least some of the motions in the one or more frames.

First claim

What is claimed is:

1 . A method comprising:
obtaining a sequence of frames captured by a video capture device during a period of time; receiving, at a machine learning system, motion sensor measurements generated by a motion sensor associated with the video capture device, the motion sensor measurements being generated based on movement of the video capture device during the period of time; based on processing the motion sensor measurements using the machine learning system, applying one or more parameters to one or more frames of the sequence of frames, the one or more parameters reducing motion in the sequence of frames resulting from the movement of the video capture device during the period of time; and generating one or more adjusted frames based on applying the one or more parameters to the one or more frames, the one or more adjusted frames having a reduction in at least some of the motion in the one or more frames.

2 . The method of claim 1 , wherein the movement of the video capture device comprises at least one of a pitch, a roll, and a yaw of the video capture device.

3 . The method of claim 2 , further comprising generating, based on the motion sensor measurements, one or more vectors representing the at least one of the pitch, the roll, and the yaw of the video capture device, wherein the one or more parameters are generated based on the one or more vectors.

4 . The method of claim 3 , further comprising:
receiving a plurality of sample motion sensor measurements; and training the machine learning system based on the plurality of sample motion sensor measurements.

5 . The method of claim 4 , wherein training the machine learning system comprises:
computing, using the machine learning system, a set of parameters for the plurality of sample motion sensor measurements; determining a degree of accuracy associated with the set of parameters; and adjusting, based on the degree of accuracy, at least one of a set of weights and a set of biases configured for the machine learning system.

6 . The method of claim 4 , wherein training of the machine learning system is activated based on operating parameters of a host of the machine learning system being within a threshold, the operating parameters including thermal, power, and computing parameters.

7 . The method of claim 1 , wherein processing the motion sensor measurements using the machine learning system includes:
classifying, using the machine learning system, patterns of motions based on the motion sensor measurements to generate one or more classified patterns of motions, wherein the one or more classified patterns of motions correlate one or more respective motions to at least one of a user associated with the video capture device and a specific category of usage of the video capture device.

8 . The method of claim 7 , wherein the specific category of usage of the video capture device comprises at least one of a first usage by the user while walking with the video capture device, a second usage by the user while running with the video capture device, a third usage by the user while standing or sitting with the video capture device, or a fourth usage by the user while traveling in a vehicle with the video capture device.

9 . The method of claim 7 , wherein the one or more parameters are based on an associated category of motions from the one or more classified patterns of motions.

10 . The method of claim 9 , wherein the one or more parameters comprise a first set of parameters based on the associated category of motions comprising a first usage by the user while walking with the video capture device, a second set of parameters based on the associated category of motions comprising a second usage by the user while running with the video capture device, a third set of parameters based on the associated category of motions comprising a third usage by the user while standing or sitting with the video capture device, and a fourth set of parameters based on the associated category of motions comprising a fourth usage by the user while traveling in a vehicle with the video capture device.

11 . The method of claim 1 , further comprising:
storing the one or more frames; receiving additional motion sensor measurements generated by the motion sensor based on additional movement of the video capture device, the additional movement of the video capture device being after the one or more frames are captured by the video capture device; and generating, using the machine learning system, the one or more parameters based at least in part on the additional motion sensor measurements.

12 . The method of claim 1 , further comprising generating the one or more parameters, wherein generating the one or more parameters comprises:
generating, using the machine learning system and the motion sensor measurements, one or more vectors representing the movement of the video capture device during the period of time; identifying, using the machine learning system, a first set of parameters for at least partially correcting one or more angle errors in a curve associated with the one or more vectors, wherein the one or more angle errors represent a delay in the curve, the first set of parameters being identified from an angle domain; and identifying, using the machine learning system, a second set of parameters for at least partially correcting one or more velocity errors in the curve associated with the one or more vectors, wherein the one or more velocity errors represent one or more ripples in the curve, the second set of parameters being identified from a velocity domain.

13 . The method of claim 1 , wherein the machine learning system includes at least one neural network.

14 . The method of claim 1 , wherein the one or more parameters stabilize the one or more frames to reduce the motion in the sequence of frames.

15 . An apparatus comprising:
a memory; and a processor configured to:
obtain a sequence of frames captured by a video capture device during a period of time;
receive, at a machine learning system, motion sensor measurements generated by a motion sensor associated with the video capture device, the motion sensor measurements being generated based on movement of the video capture device during the period of time;
based on processing the motion sensor measurements using the machine learning system, applying one or more parameters to one or more frames of the sequence of frames, the one or more parameters reducing motion in the sequence of frames resulting from the movement of the video capture device during the period of time; and
generate one or more adjusted frames based on applying the one or more parameters to the one or more frames, the one or more adjusted frames having a reduction in at least some of the motion in the one or more frames.

16 . The apparatus of claim 15 , wherein the movement of the video capture device comprises at least one of a pitch, a roll, and a yaw of the video capture device.

17 . The apparatus of claim 16 , wherein the processor is configured to: generate, based on the motion sensor measurements, one or more vectors representing the at least one of the pitch, the roll, and the yaw of the video capture device, wherein the one or more parameters are generated based on the one or more vectors.

18 . The apparatus of claim 17 , wherein the processor is configured to:
receive a plurality of sample motion sensor measurements; and train the machine learning system based on the plurality of sample motion sensor measurements.

Assignees

• Qualcomm Inc

Inventors

• Kang Young Hoon
• Park Hee Jun
• Kazi Tauseef
• Gaizman Ron
• Pinhasov Eran
• Tzur Meir

Classifications

• H04N23/64

  Computer-aided capture of images, e.g. transfer from script file into camera, check of taken image quality, advice or proposal for image composition or decision on when to take image · CPC title

• G06N3/044

  Recurrent networks, e.g. Hopfield networks · CPC title

• H04N23/6812

  based on additional sensors, e.g. acceleration sensors · CPC title

• H04N23/63

  by using electronic viewfinders · CPC title

• H04N23/683 Primary

  performed by a processor, e.g. controlling the readout of an image memory · CPC title