Methods and apparatus for electronic image stabilization based on a lens polynomial

Patent metadata

Field	Value
Publication number	US-12262115-B2
Application number	US-202217804661-A
Country	US
Kind code	B2
Filing date	May 31, 2022
Priority date	Jan 28, 2022
Publication date	Mar 25, 2025
Grant date	Mar 25, 2025

Abstract

Methods and apparatus for stabilizing image data based on a lens polynomial. Non-rectilinear footage can be captured and rectified in-camera; the rectified images may be stabilized to provide rectified stable video. In one exemplary embodiment, the footage is rectified and stabilized based on a lens polynomial and the camera's own movement. In some variants, the rectified stable video may be stored along with its margin track. In-camera rectified stable video provides several benefits over traditional techniques (e.g., the ability to share rectilinear content from the camera without additional post-processing, as well as reduced file sizes of the shared videos). Lens-aware post-processing can reuse portions of the in-camera rectified stable videos while providing additional benefits (e.g., the ability to re-frame the video in post-production).

First claim

What is claimed is:

1. A post-processing apparatus, comprising:
a processor subsystem; and
a non-transitory computer-readable medium that stores instructions which when executed by the processor subsystem, causes the post-processing apparatus to:
obtain a video comprising previously stabilized rectilinear image data according to an originally captured view;
obtain a margin track that corresponds to the previously stabilized rectilinear image data;
obtain telemetry data and a lens characteristic;
determine a new view based on the previously stabilized rectilinear image data and at least one of the margin track, the telemetry data, and the lens characteristic;
correct a first portion of the previously stabilized rectilinear image data and a second portion of the margin track based on the lens characteristic;
stabilize the first portion of the previously stabilized rectilinear image data and the second portion of the margin track based on the telemetry data; and
stitch the first portion and the second portion into a video frame.

2. The post-processing apparatus of claim 1 , further comprising a codec that is configured to approximate image motion based on straight-line motion vectors and address image data based on row and column addressing; and
where the instructions are further configured to cause the post-processing apparatus to encode the video frame into a re-framed rectilinear stabilized video.

3. The post-processing apparatus of claim 2 , further comprising a display, and where the instructions, when executed by the processor subsystem, further causes the post-processing apparatus to display the re-framed rectilinear stabilized video.

4. The post-processing apparatus of claim 2 , further comprising a network interface, and where the instructions, when executed by the processor subsystem, further causes the post-processing apparatus to transfer the re-framed rectilinear stabilized video to another device for display.

5. The post-processing apparatus of claim 1 , further comprising a network interface, and where the instructions, when executed by the processor subsystem, further causes the post-processing apparatus to obtain the previously stabilized rectilinear image data from another device that is streaming the video.

6. The post-processing apparatus of claim 1 , further comprising a user interface that enables a user to identify a subject-of-interest within a frame of the video; and
where the instructions, when executed by the processor subsystem, further causes the post-processing apparatus to determine the new view based on the subject-of-interest.

7. A method, comprising:
obtaining a video comprising previously stabilized rectilinear image data according to an originally captured view;
obtaining a margin track that corresponds to the previously stabilized rectilinear image data;
obtaining telemetry data and a lens characteristic;
determining a new view based on the previously stabilized rectilinear image data and at least one of the margin track, the telemetry data, and the lens characteristic;
correcting a first portion of the previously stabilized rectilinear image data and a second portion of the margin track based on the lens characteristic;
stabilizing the first portion of the previously stabilized rectilinear image data and the second portion of the margin track based on the telemetry data; and
stitching the first portion and the second portion into a video frame.

8. The method of claim 7 , further encoding the video frame into a re-framed rectilinear stabilized video.

9. The method of claim 8 , further comprising displaying the re-framed rectilinear stabilized video.

10. The method of claim 8 , further comprising transferring the re-framed rectilinear stabilized video to another device for display.

11. The method of claim 7 , further comprising obtaining the previously stabilized rectilinear image data from another device that is streaming the video.

12. The method of claim 7 , further comprising identifying a subject-of-interest within a frame of the video and determining the new view based on the subject-of-interest.

13. A method, comprising:
obtaining a video, a margin track, a telemetry track, and a lens characteristic that corresponds to the video;
determining a view based on the video and at least one of the margin track, the telemetry track, and the lens characteristic;
correcting a first frame of the video and a margin portion of the margin track based on the lens characteristic;
stabilizing the first frame of the video and the margin portion of the margin track based on the telemetry track; and
stitching the first frame and the margin portion into a new video frame.

14. The method of claim 13 , where the video comprises a previously stabilized video.

15. The method of claim 13 , where the video, the margin track, and the telemetry track, are obtained from a shared data structure.

16. The method of claim 13 , where at least one of the video, the margin track, and the telemetry track, are obtained from different data structures.

17. The method of claim 13 , where the video, the margin track, and the telemetry track, are obtained from a ready-to-share data stream.

18. The method of claim 13 , where the view is based on user input.

19. The method of claim 13 , where the view is based on the telemetry track.

20. The method of claim 19 , where the telemetry track comprises a flag that identifies an error in stabilization.

Assignees

• Gopro Inc

Inventors

• Stimm Daryl
• Schwartz Kyler William
• Russell Andrew
• Mcintosh Robert

Classifications

• H04N23/6812

  based on additional sensors, e.g. acceleration sensors · CPC title

• H04N23/6811

  based on the image signal · CPC title

• H04N23/698

  for achieving an enlarged field of view, e.g. panoramic image capture · CPC title

• H04N23/45 Primary

  for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images · CPC title

• H04N23/683 Primary

  performed by a processor, e.g. controlling the readout of an image memory · CPC title