Hybrid zoom systems and methods for improved electronic image stabilization of unmanned aerial vehicle video streams

Patent metadata

Field	Value
Publication number	US-12538029-B2
Application number	US-202318523003-A
Country	US
Kind code	B2
Filing date	Nov 29, 2023
Priority date	Nov 29, 2022
Publication date	Jan 27, 2026
Grant date	Jan 27, 2026

Abstract

Systems and methods for operating an unmanned aerial vehicle (UAV). A method includes capturing a raw video stream input using one or more camera sensors of the UAV. The method includes determining a zoom level for the raw video stream input. For the zoom level being determined to be greater than 3×, the method includes: first zooming the raw video stream input to a first field of view (FOV) at 3× to generate a 3× zoomed video stream; second zooming the 3× zoomed video stream to a second FOV at a second zoom level to generate a fully zoomed video stream at greater than 3×; and applying electronic image stabilization (EIS) to the fully zoomed video stream to generate a zoomed and stabilized video stream. Embodiments of the present technology provide a hybrid zoom approach with improved EIS performance to enhance the UAV user experience.

First claim

What is claimed is:

1 . A method of operating an unmanned aerial vehicle (UAV), the method comprising:
capturing a raw video stream input using one or more camera sensors of the UAV; determining a zoom level for the raw video stream input; and responsive to determining the zoom level being less than or equal to 3×,
zooming the raw video stream input to a field of view at a first zoom level of less than or equal to 3× to generate a zoomed video stream; and
applying electronic image stabilization to the zoomed video stream to generate a zoomed and stabilized video stream; and
responsive to determining the zoom level being greater than 3×,
first zooming the raw video stream input to a first field of view at 3× to generate a 3× zoomed video stream;
second zooming the 3× zoomed video stream to a second field of view at a second zoom level to generate a fully zoomed video stream at greater than 3×; and
applying electronic image stabilization to the fully zoomed video stream to generate a zoomed and stabilized video stream.

2 . The method of claim 1 further comprising establishing a communications link between the UAV and a ground station.

3 . The method of claim 2 further comprising receiving, by the UAV via the communications link, a signal encoding data representative of the zoom level from the ground station.

4 . The method of claim 3 , wherein determining the zoom level for the raw video stream input comprises determining the zoom level according to the data representative of the zoom level.

5 . The method of claim 1 further comprising encoding the zoomed and stabilized video stream to generate a zoomed, stabilized and encoded video stream.

6 . The method of claim 5 further comprising scaling the zoomed and stabilized video stream to generate an unencoded, zoomed, scaled and stabilized video stream.

7 . The method of claim 6 , wherein encoding the zoomed and stabilized video stream comprises encoding the unencoded, zoomed, scaled and stabilized video stream.

8 . The method of claim 5 further comprising transmitting the zoomed, stabilized and encoded video stream to a ground station communicably coupled to the UAV via a communications link.

9 . The method of claim 1 further comprising:
detecting an acceleration of the one or more camera sensors; and
computing a vector representative of the acceleration of the one or more camera sensors.

10 . The method of claim 9 , wherein applying electronic image stabilization to the zoomed video stream comprises:
producing a stabilization viewport inside a perimeter of the field of view, wherein the stabilization viewport and the perimeter of the field of view are separated by a margin; computing an opposing vector counteracting the vector; and moving the stabilization viewport within the margin according to the opposing vector to generate the zoomed and stabilized video stream.

11 . The method of claim 10 , wherein producing the stabilization viewport comprises cropping the field of view by a predetermined percentage defining the margin.

12 . The method of claim 10 further comprising generating mesh data defining coordinates of the stabilization viewport within the field of view according to the vector and at least one property of a lens of the one or more camera sensors, wherein moving the stabilization viewport comprises moving the stabilization viewport further according to the mesh data.

13 . The method of claim 12 , wherein producing the stabilization viewport comprises generating a texture map for the stabilization viewport according to the mesh data, and wherein moving the stabilization viewport comprises moving the stabilization viewport further according to the texture map.

14 . The method of claim 10 further comprising transmitting the zoomed and stabilized video stream as the stabilization viewport to a ground station communicably coupled to the UAV via a communications link.

15 . The method of claim 9 , wherein applying electronic image stabilization to the fully zoomed video stream comprises:
producing a stabilization viewport inside a perimeter of the second field of view, wherein the stabilization viewport and the perimeter of the second field of view are separated by a margin; computing an opposing vector counteracting the vector; and moving the stabilization viewport within the margin according to the opposing vector to generate the zoomed and stabilized video stream.

16 . The method of claim 15 , wherein producing the stabilization viewport comprises cropping the second field of view by a predetermined percentage defining the margin.

17 . The method of claim 15 further comprising generating mesh data defining coordinates of the stabilization viewport within the second field of view according to the vector and at least one property of a lens of the one or more camera sensors, wherein moving the stabilization viewport comprises moving the stabilization viewport further according to the mesh data.

18 . The method of claim 17 , wherein producing the stabilization viewport comprises generating a texture map for the stabilization viewport according to the mesh data, and wherein moving the stabilization viewport comprises moving the stabilization viewport further according to the texture map.

19 . The method of claim 15 further comprising transmitting the zoomed and stabilized video stream as the stabilization viewport to a ground station communicably coupled to the UAV via a communications link.

20 . A system for operating an unmanned aerial vehicle (UAV), the system comprising:
one or more camera sensors for capturing a raw video stream input; and at least one computing device disposed onboard the UAV and operably coupled to the one or more camera sensors, the at least one computing device configured to:
determine a zoom level for the raw video stream input;
responsive to determining the zoom level being less than or equal to 3×,
zoom the raw video stream input to a field of view at a first zoom level of less than or equal to 3× to generate a zoomed video stream; and
apply electronic image stabilization to the zoomed video stream to generate a zoomed and stabilized video stream; and

responsive to determining the zoom level being greater than 3×,
first zoom the raw video stream input to a first field of view at 3× to generate a 3× zoomed video stream;
second zoom the 3× zoomed video stream to a second field of view at a second zoom level to generate a fully zoomed video stream at greater than 3×; and
apply electronic image stabilization to the fully zoomed video stream to generate a zoomed and stabilized video stream.

21 . One or more non-transitory computer readable media having stored thereon program instructions which, when executed by at least one processor, cause an unmanned aerial vehicle (UAV) to:
determine a zoom level for a raw video stream input captured by one or more camera sensors of the UAV; responsive to determining the zoom level being less than or equal to 3×,
zoom the raw video stream input to a field of view at a first zoom level of less than or equal to 3× to generate a zoomed video stream; and
apply electronic image stabilization to the zoomed video stream to generate a zoomed and stabilized video stream; and
responsive to determining the zoom level being greater than 3×,
first zoom the raw video stream input to a first field of view at 3× to generate a 3× zoomed video stream;
second zoom the 3× zoomed video stream to a second fiel

Assignees

• Skydio Inc

Inventors

• Moskovchenko Stepan

Classifications

• H04N23/683

  performed by a processor, e.g. controlling the readout of an image memory · CPC title

• G06V20/17

  taken from planes or by drones · CPC title

• G06T3/40 Primary

  Scaling of whole images or parts thereof, e.g. expanding or contracting · CPC title

• H04N23/69 Primary

  Control of means for changing angle of the field of view, e.g. optical zoom objectives or electronic zooming · CPC title

• H04N23/6812

  based on additional sensors, e.g. acceleration sensors · CPC title