Methods and apparatus for maximizing codec bandwidth in video applications

Patent metadata

Field	Value
Publication number	US-11800141-B2
Application number	US-202117453217-A
Country	US
Kind code	B2
Filing date	Nov 2, 2021
Priority date	Jun 26, 2019
Publication date	Oct 24, 2023
Grant date	Oct 24, 2023

Abstract

Methods and apparatus for processing of video content to optimize codec bandwidth. In one embodiment, the method includes capturing panoramic imaging content (e.g., a 360° panorama), mapping the panoramic imaging content into an equi-angular cubemap (EAC) format, and splitting the EAC format into segments for transmission to maximize codec bandwidth. In one exemplary embodiment, the EAC segments are transmitted at a different frame rate than the subsequent display rate of the panoramic imaging content. For example, the mapping and frame rate may be chosen to enable the rendering of 8K, 360° content at 24 fps, using commodity encoder hardware and software that nominally supports 4K content at 60 fps.

First claim

What is claimed is:

1. A method for piecemeal encoding of a video frame, comprising:
capturing the video frame at a frame rate;
splitting the video frame into a plurality of portions, where each portion comprises adjacent facets that share an edge of a cubemap to reduce compression artifacts caused by image discontinuities;
selecting a number of codecs to power based on a time multiplex ratio of an encoding rate to the frame rate;
encoding the plurality of portions at the encoding rate to generate a plurality of encoded segments; and
transmitting the plurality of encoded segments to a display device.

2. The method of claim 1 , where capturing the video frame comprises:
capturing a first image via a first camera and a second image via a second camera; and
stitching the first image and the second image.

3. The method of claim 2 , where stitching the first image and the second image further comprises:
identifying redundant image data in the first image and the second image; and
stretching or shrinking pixels of the first image or the second image at a stitch line based on the redundant image data.

4. The method of claim 3 , further comprising transmitting a warp track to the display device that describes the pixels of the first image or the second image that were stretched or shrunk.

5. The method of claim 2 , where stitching the first image and the second image further comprises projecting the first image and the second image to a cubemap projection comprising a plurality of facets.

6. The method of claim 1 , where the plurality of encoded segments are transmitted to the display device at the encoding rate; and
enabling the display device to display the video frame at the frame rate.

7. A capture apparatus configured to piecemeal encode video frames, comprising:
a first camera;
a plurality of codecs;
a processor; and
a non-transitory computer readable medium comprising instructions that when executed by the processor, cause the capture apparatus to:
capture a video frame at a frame rate;
split the video frame into a plurality of portions;
select a number of codecs from the plurality of codecs to power based on a time multiplex ratio of an encoding rate to the frame rate;
encode the plurality of portions at the encoding rate to generate a plurality of encoded segments; and
transmit the plurality of encoded segments to a display device.

8. The capture apparatus of claim 7 , further comprising a second camera;
where the instructions, when executed by the processor, cause the capture apparatus to:
capture a first image via the first camera and a second image via the second camera; and
stitch the first image and the second image to generate the video frame.

9. The capture apparatus of claim 7 , where the video frame is a cubemap projection of a plurality of facets and each portion of the plurality of portions comprises at least two adjacent facets of the plurality of facets.

10. The capture apparatus of claim 9 , where the instructions, when executed by the processor, cause the capture apparatus to:
encode a first portion of the plurality of portions at a first time interval; and
encode a second portion of the plurality of portions at a second time interval after the first time interval.

11. A display apparatus configured to piecemeal decode video frames, comprising:
a processing system comprising:
a codec;
a shared memory buffer;
a central processing unit coupled to the shared memory buffer;
a graphic processing unit coupled to the shared memory buffer; and

a non-transitory computer readable medium comprising instructions that when executed by the processing system, cause the display apparatus to:
receive a plurality of encoded segments;
decode the plurality of encoded segments at a decoding rate to generate a plurality of portions;
write a first portion of the plurality of portions to a first location of the shared memory buffer and write a second portion of the plurality of portions to a second location of the shared memory buffer;
combine the plurality of portions into a video frame; and
render the plurality of portions into the video frame at a frame rate that is a time multiplex of the decoding rate.

12. The display apparatus of claim 11 , where the instructions, when executed by the processor processing system, cause the display apparatus to:
write a third portion of the plurality of portions to a third location of the shared memory buffer.

13. The display apparatus of claim 11 , further comprising a display and the instructions, when executed by the processing system, cause the display apparatus to display the video frame.

14. The display apparatus of claim 13 , where the video frame comprises a cubemap projection of a plurality of facets and each portion of the plurality of portions comprises at least two adjacent facets of the plurality of facets.

15. The display apparatus of claim 14 , where the cubemap projection has a horizontal pixel width of 5376 pixels and the frame rate comprises 30 frames per second.

16. The display apparatus of claim 15 , where the cubemap projection can provide 120° panoramas in any direction at a resolution of 1920×1080.

17. The display apparatus of claim 14 , where the cubemap projection has a horizontal pixel width of 7680 pixels and the frame rate comprises 24 frames per second.

18. The display apparatus of claim 14 , where the cubemap projection has a horizontal pixel width of 6848 pixels and the frame rate comprises 30 frames per second.

Assignees

• Gopro Inc

Inventors

• Newman David A
• Douady Cesar
• Lefebvre Alexis
• Stimm Daryl
• Kraak Michael

Classifications

• H04N19/52 Primary

  by predictive encoding · CPC title

• G06T3/0087

  Physics · mapped topic

• H04N19/182

  the unit being a pixel · CPC title

• H04N19/198

  including smoothing of a sequence of encoding parameters, e.g. by averaging, by choice of the maximum, minimum or median value · CPC title

• H04N19/40

  using video transcoding, i.e. partial or full decoding of a coded input stream followed by re-encoding of the decoded output stream · CPC title