Methods and apparatus for re-stabilizing video in post-processing

Patent metadata

Field	Value
Publication number	US-12477223-B2
Application number	US-202418670673-A
Country	US
Kind code	B2
Filing date	May 21, 2024
Priority date	Oct 1, 2021
Publication date	Nov 18, 2025
Grant date	Nov 18, 2025

Abstract

Methods and apparatus for post-processing in-camera stabilized video. Embodiments of the present disclosure reconstruct and re-stabilize an in-camera stabilized video to provide for improved stabilization (e.g., a wider crop, etc.). In-camera sensor data may be stored and used to re-calculate orientation metadata in post-production. In-camera stabilization provides several benefits (e.g., the ability to share stabilized videos from the camera without additional post-processing as well as reduced file sizes of the shared videos). Camera-aware post-processing can reuse portions of the in-camera stabilized videos while providing additional benefits (e.g., the ability to regenerate the original captured videos in post-production and re-stabilize the videos). Camera-aware post-processing can also improve orientation metadata and remove sensor error. The disclosed techniques also enable assisted optical flow-based stabilization using the refined metadata.

First claim

What is claimed is:

1 . A method for post-processing video, comprising:
receiving video data and margin data associated with the video data; reconstructing a set of previously stabilized images from the video data and the margin data; receiving a designation of a frame of the video data as a keyframe; re-stabilizing the set of previously stabilized images based on the designation of the frame as the keyframe creating re-stabilized video; and encoding the re-stabilized video including at least a portion of the margin data creating encoded re-stabilized video.

2 . The method of claim 1 , where receiving the designation of the frame as the keyframe is based on a user selection.

3 . The method of claim 1 , where the keyframe indicates a fixed reference for inclusion during the re-stabilizing the previously stabilized images.

4 . The method of claim 1 , further comprising receiving a second designation of a second frame of the video data as a second keyframe, where re-stabilizing the set of previously stabilized images is further based on the second designation of the second frame as the second keyframe.

5 . The method of claim 1 , where receiving the video data and the margin data comprise receiving a video file comprising a video track comprising the video data and a margin data track comprising the margin data.

6 . The method of claim 5 , where the video file further comprises a metadata track comprising orientation metadata associated with the video data.

7 . The method of claim 6 , further comprising updating the metadata track based on determining at least one of an image orientation, a camera orientation, a gravity vector, an accelerometer value, and a gyroscope value.

8 . The method of claim 6 , further comprising:
obtaining a captured trajectory of a camera from the metadata track; and obtaining a designated trajectory of the video track from the metadata track.

9 . The method of claim 8 , further comprising updating the designated trajectory based on the keyframe.

10 . The method of claim 1 , further comprising rendering the encoded re-stabilized video for viewing.

11 . A camera apparatus, comprising:
a camera; an orientation sensor; a processor; and a non-transitory computer-readable medium that stores instructions when executed by the processor, causes the camera apparatus to:
capture a video comprising a series of frames;
determine a plurality of camera orientations based on the orientation sensor for the series of frames;
determine a plurality of corresponding frame orientations to counteract the plurality of camera orientations for the series of frames;
crop the series of frames based on the plurality of corresponding frame orientations to create a cropped video having a designated view and margin data;
encode the cropped video within a video track of a video file;
encode the margin data within a margin track of the video file; and
encode the plurality of camera orientations and the plurality of corresponding frame orientations within an orientation metadata track of the video file.

12 . The camera apparatus of claim 11 , where the instructions when executed by the processor, further causes the camera apparatus to receive a resolution selection for the designated view.

13 . The camera apparatus of claim 11 , where the instructions when executed by the processor, further causes the camera apparatus to determine whether to encode the video file optimized for viewing on the camera apparatus or a separate display device.

14 . The camera apparatus of claim 13 , where:
the camera apparatus determines to encode the video file optimized for viewing on the separate display device, and the instructions when executed by the processor, further causes the camera apparatus to send the video file to the separate display device for display.

15 . The camera apparatus of claim 13 , where:
the camera apparatus determines to encode the video file optimized for viewing on the separate display device, and the instructions when executed by the processor, further causes the camera apparatus to send the video track to the separate display device for display.

16 . The camera apparatus of claim 11 , where the video file is generated according to a real-time budget.

17 . A camera-aware post-processing apparatus, comprising:
a processor; and a non-transitory computer-readable medium that stores instructions which when executed by the processor, causes the camera-aware post-processing apparatus to:
obtain a margin data and an orientation metadata corresponding to a video data with a first designated view size;
reconstruct a set of previously stabilized frames from the video data and the margin data based on the orientation metadata; and
generate a re-stabilized video from the set of previously stabilized frames with a second designated view size having a resolution larger of at least one dimension than the first designated view size.

18 . The camera-aware post-processing apparatus of claim 17 , further comprising a codec configured to decode video files, where:
the instructions, when executed by the processor, cause the camera-aware post-processing apparatus to decode the video data from a video track of a multi-track video data structure, the margin data from a margin track of the multi-track video data structure, and the orientation metadata from an orientation track of the multi-track video data structure.

19 . The camera-aware post-processing apparatus of claim 17 , where:
the orientation metadata comprises capture orientation metadata and stabilization orientation metadata, and reconstructing the set of previously stabilized frames comprises manipulating the video data based on differences between the stabilization orientation metadata and the capture orientation metadata.

20 . The camera-aware post-processing apparatus of claim 17 , where the instructions, when executed by the processor, cause the camera-aware post-processing apparatus to:
estimate motion between at least two images of the set of previously stabilized frames using an optical flow technique, and re-stabilize the set of previously stabilized frames based on the motion.

Assignees

• Gopro Inc

Inventors

• Stimm Daryl
• Russell Andrew
• Mcintosh Robert

Classifications

• H04N23/6812

  based on additional sensors, e.g. acceleration sensors · CPC title

• H04N23/6811 Primary

  based on the image signal · CPC title

• H04N23/80

  Camera processing pipelines; Components thereof · CPC title

• H04N7/0137

  dependent on presence/absence of motion, e.g. of motion zones (H04N7/014 takes precedence; movement detection in television signals H04N5/144) · CPC title

• H04N23/683 Primary

  performed by a processor, e.g. controlling the readout of an image memory · CPC title