Methods and apparatus for re-stabilizing video in post-processing

Patent metadata

Field	Value
Publication number	US-12035044-B2
Application number	US-202117449713-A
Country	US
Kind code	B2
Filing date	Oct 1, 2021
Priority date	Oct 1, 2021
Publication date	Jul 9, 2024
Grant date	Jul 9, 2024

Abstract

Methods and apparatus for post-processing in-camera stabilized video. Embodiments of the present disclosure reconstruct and re-stabilize an in-camera stabilized video to provide for improved stabilization (e.g., a wider crop, etc.) In-camera sensor data may be stored and used to re-calculate orientation metadata in post-production. In-camera stabilization provides several benefits (e.g., the ability to share stabilized videos from the camera without additional post-processing as well as reduced file sizes of the shared videos). Camera-aware post-processing can reuse portions of the in-camera stabilized videos while providing additional benefits (e.g., the ability to regenerate the original captured videos in post-production and re-stabilize the videos). Camera-aware post-processing can also improve orientation metadata and remove sensor error. The disclosed techniques also enable assisted optical flow-based stabilization using the refined metadata.

First claim

What is claimed is:

1. A method for post-processing video, comprising:
receiving a video track, an orientation metadata track, and a margin track;
reconstructing a set of previously stabilized images from the video track and the margin track;
determining a reusable portion of the set of previously stabilized images based on the orientation metadata track;
updating the orientation metadata track based on the reusable portion; and
re-stabilizing the reusable portion based on the updated orientation metadata track to generate a re-stabilized video.

2. The method of claim 1 , where updating the orientation metadata track comprises determining at least one of an image orientation, a camera orientation, a gravity vector, an accelerometer value, and a gyroscope value.

3. The method of claim 1 , further comprising re-encoding the re-stabilized video; and where the re-stabilized video includes data from the margin track.

4. The method of claim 1 , where re-stabilizing the reusable portion comprises cropping the set of previously stabilized images based on the updated orientation metadata track.

5. The method of claim 4 , where the reusable portion of the set of previously stabilized images is determined based on an optical flow-based stabilization.

6. The method of claim 5 , where the margin track comprises portions of the set of previously stabilized images that were cropped out based on a sensor-based stabilization.

7. The method of claim 1 , where the video track, the orientation metadata track, and the margin track are received according to a real-time budget or a near real-time budget.

8. The method of claim 1 , where the reusable portion of the set of previously stabilized images is determined with according to a best-effort budget.

9. The method of claim 1 , where the orientation metadata track comprises an error flag that indicates an image of the set of previously stabilized images that was incorrectly oriented.

10. The method of claim 9 , where updating the orientation metadata track comprises correcting an orientation of the image.

11. An apparatus, comprising:
a processor; and
a non-transitory computer-readable medium that stores instructions which when executed by the processor, causes the apparatus to:
receive a video track, an orientation metadata track, and a margin track;
reconstruct a set of previously stabilized images from the video track and the margin track;
determine a reusable portion of the set of previously stabilized images based on the orientation metadata track;
update the orientation metadata track based on the reusable portion; and
re-stabilize the reusable portion based on the updated orientation metadata track to generate a re-stabilized video.

12. The apparatus of claim 11 , where the instructions, when executed by the processor, cause the apparatus to determine at least one of an image orientation, a camera orientation, a gravity vector, an accelerometer value, and a gyroscope value.

13. The apparatus of claim 11 , where the instructions, when executed by the processor, cause the apparatus to re-encode the re-stabilized video, where the re-stabilized video includes data from the margin track.

14. The apparatus of claim 11 , where the instructions, when executed by the processor, cause the apparatus to crop the set of previously stabilized images based on the updated orientation metadata track.

15. The apparatus of claim 14 , where the reusable portion of the set of previously stabilized images is determined based on an optical flow-based stabilization.

16. The apparatus of claim 15 , where the margin track comprises portions of the set of previously stabilized images that were cropped out based on a sensor-based stabilization.

17. The apparatus of claim 11 , where the video track, the orientation metadata track, and the margin track are received according to a real-time budget or a near real-time budget.

18. The apparatus of claim 11 , where the reusable portion of the set of previously stabilized images is determined according to a best-effort budget.

19. The apparatus of claim 11 , where the orientation metadata track comprises an error flag that indicates an image of the set of previously stabilized images that was incorrectly oriented.

20. The apparatus of claim 19 , where the instructions, when executed by the processor, cause the apparatus to correct an orientation of the image.

Assignees

• Gopro Inc

Inventors

• Stimm Daryl
• Russell Andrew
• Mcintosh Robert

Classifications

• H04N23/80

  Camera processing pipelines; Components thereof · CPC title

• H04N23/6812

  based on additional sensors, e.g. acceleration sensors · CPC title

• H04N23/6811 Primary

  based on the image signal · CPC title

• H04N7/0137

  dependent on presence/absence of motion, e.g. of motion zones (H04N7/014 takes precedence; movement detection in television signals H04N5/144) · CPC title

• H04N23/683 Primary

  performed by a processor, e.g. controlling the readout of an image memory · CPC title