Apparatus and methods for providing a cubic transport format for multi-lens spherical imaging

Patent metadata

Field	Value
Publication number	US-11330245-B2
Application number	US-202016823118-A
Country	US
Kind code	B2
Filing date	Mar 18, 2020
Priority date	Jul 19, 2016
Publication date	May 10, 2022
Grant date	May 10, 2022

Abstract

Apparatus and methods for providing a frame packing arrangement for the encoding/decoding of, for example, panoramic content. In one embodiment, the frame packing arrangement utilizes overlapping imaging data so as to enable, for example, a post-decode stitching operation to be performed. The frame packing arrangement may utilize a number of projection formats, such as a cubemap projection, and may utilize any number of differing aspect ratios such as, without limitation, 4×3, 3×2, 4×2, 2×4 aspect ratios. Additionally, the overlapping imaging data may be positioned within the frame packing arrangement chosen so as to improve upon coding efficiency. For example, the overlapping imaging data may be positioned within the frame packing arrangement so as to emphasize image continuity. An encoder apparatus and decoder apparatus for use with the aforementioned frame packing arrangements are also disclosed.

First claim

What is claimed:

1. A computerized method for rendering an image, comprising:
obtaining encoded image data in a transport format, the encoded image data comprising a stitch line, a plurality of image facets associated with the stitch line, and one or more overlapping portions associated with at least a portion of the plurality of image facets, where the stitch line, the plurality of image facets, and the one or more overlapping portions are configured to be in a prescribed arrangement within the transport format;
decoding at least the plurality of image facets to obtain a decoded plurality of facets;
responsive to a determination not to discard the one or more overlapping portions, performing a stitching operation on the decoded plurality of facets along the stitch line, the stitching operation producing a stitched image; and
causing the stitched image to be rendered for display.

2. The computerized method of claim 1 , wherein the determination whether to discard the one or more overlapping portions is based at least on availability of one or more computing resources, further comprising:
based at least on a determination to discard the at least one overlapping portion, applying a cut operation along the stitch line for the stitching operation, the performing of the stitching operation comprising the applying of the cut operation.

3. The computerized method of claim 1 , further comprising:
determining whether to discard at least one of the one or more overlapping portions based at least on availability of one or more computing resources;
decoding the one or more overlapping portions; and
based at least on a determination not to discard the at least one overlapping portion, applying a full-stitch operation along the stitch line for the stitching operation, the stitching operation including a first stitch on the decoded plurality of facets and a second stitch on the decoded one or more overlapping portions.

4. The computerized method of claim 1 , wherein the plurality of image facets comprise non-overlapping portions.

5. The computerized method of claim 1 , wherein:
the encoded image data further comprises one or more untouched image facets, the one or more untouched image facets comprising facets captured by a single lens of a multi-lens image capture device; and
the computerized method further comprises decoding the untouched image facets.

6. The computerized method of claim 1 , wherein:
the transport format comprises a frame packing arrangement, the frame packing arrangement comprising at least a cubic projection, each of a subset of the decoded plurality of facets having been encoded from (i) a respective portion of an image captured by a first image capture device, and (ii) a respective portion of an image captured by a second image capture device.

7. The computerized method of claim 6 , wherein the frame packing arrangement further comprises an additional image facet separate from the cubic projection, the additional image facet comprising the one or more overlapping portions.

8. A non-transitory computer-readable apparatus comprising a storage medium, the storage medium comprising a computer program having a plurality of instructions configured to, when executed by a processor apparatus of a computerized device, cause the computerized device to:
receive an encoded transport data structure comprising encoded image data, the encoded image data comprising image facets and overlapping portions associated with the image facets, the encoded transport data structure comprising a cubic projection of a panoramic image;
decode at least two of the image facets to obtain decoded image facets, and decode corresponding ones of the overlapping portions to obtain decoded corresponding ones of the overlapping portions;
responsive to a determination not to discard the overlapping portions, generate a stitched image via application of a stitch process to the decoded image facets and to the decoded corresponding ones of the overlapping portions; and
cause rendering of the stitched image on a display.

9. The non-transitory computer-readable apparatus of claim 8 , wherein the panoramic image comprises (i) a first image captured via a first spherical lens and (ii) a second image captured via a second spherical lens.

10. The non-transitory computer-readable apparatus of claim 9 wherein:
the encoded transport data structure comprises a frame packing arrangement, the frame packing arrangement comprising a first plurality of the image facets arranged in a first direction and a second plurality of the image facets arranged in a second direction; and
the first plurality of the image facets comprises (i) a portion of the encoded image data captured by the first spherical lens and (ii) a portion of the encoded image data captured by the second spherical lens.

11. The non-transitory computer-readable apparatus of claim 10 , wherein the frame packing arrangement further comprises a separate imaging facet, the separate imaging facet comprising the overlapping portions associated with the image facets.

12. The non-transitory computer-readable apparatus of claim 10 , wherein the frame packing arrangement further comprises the overlapping portions associated with the image facets, the overlapping portions being placed between (i) a corresponding image portion associated with the portion of the encoded image data captured by the first spherical lens and (ii) a corresponding image portion associated with the portion of the encoded image data captured by the second spherical lens.

13. The non-transitory computer-readable apparatus of claim 10 , wherein the frame packing arrangement further comprises the overlapping portions associated with the image facets, the overlapping portions being placed within one or more of (i) an image portion associated with the portion of the encoded image data captured by the first spherical lens and (ii) an image portion associated with the portion of the encoded image data captured by the second spherical lens.

14. A computerized method for decoding image data, comprising:
receiving a data structure comprising encoded image data, the encoded image data comprising a plurality of image facets and a plurality of overlapping portions associated with respective ones of the plurality of image facets arranged in a given transport format within the data structure;
decoding at least a portion of the plurality of image facets and at least a portion of the plurality of overlapping portions;
responsive to a determination not to discard the plurality of overlapping portions, generating a stitched image, the generating comprising:
(i) algorithmically stitching together the decoded at least portion of the plurality of image facets, and
(ii) algorithmically stitching together the decoded at least portion of the plurality of overlapping portions; and

causing the stitched image to be rendered for display.

15. The computerized method of claim 14 , wherein the data structure comprises a frame packing arrangement for the plurality of image facets and the plurality of overlapping portions, the frame packing arrangement comprising an aspect ratio selected from the group consisting of 4×3, 4×2, 2×4, and 3×2, the aspect ratio enabling continuity between at least a portion of the plurality of image facets and at least a portion of the plurality of overlapping portions.

16. The computerized method of claim 14 , wherein the generating of the stitched image comprises applying a full-stitch operation subsequent to a determination to keep the at least portion of the plurality of overlapping portions.

Assignees

• Gopro Inc

Inventors

• Abbas Adeel
• Macmillan Timothy

Classifications

• H04N13/243 Primary

  using three or more two-dimensional [2D] image sensors · CPC title

• H04N23/698

  for achieving an enlarged field of view, e.g. panoramic image capture · CPC title

• G06F18/22

  Matching criteria, e.g. proximity measures · CPC title

• G06T2200/04

  involving 3D image data · CPC title

• H04N13/161

  Encoding, multiplexing or demultiplexing different image signal components (for multi-view video sequence encoding H04N19/597) · CPC title