Methods and apparatus for shared image processing among multiple devices

Patent metadata

Field	Value
Publication number	US-11734796-B2
Application number	US-202117230135-A
Country	US
Kind code	B2
Filing date	Apr 14, 2021
Priority date	Apr 15, 2020
Publication date	Aug 22, 2023
Grant date	Aug 22, 2023

Abstract

Methods and apparatus for shared image processing among multiple devices. In one embodiment, an exemplary action camera performs a partial multiband blend. Even though the action camera may not have resources to handle the multiband blend of the entire action camera's footage, it can do a significant portion. The partially blended content can be used in ready-to-share applications, or completely blended by another device.

First claim

What is claimed is:

1. A method for shared image processing, the method comprising:
obtaining spherical content;
converting the spherical content to an overlapping data structure;
performing, by a first device, a first portion of a multiband blend operation on the overlapping data structure generating a first image processed data structure; and
transferring the first image processed data structure to a second device.

2. The method for shared image processing of claim 1 , wherein the spherical content comprises a plurality of hyper-hemispherical content with at least one overlapping portion between the plurality of hyper-hemispherical content to enable a device to stitch the plurality of hyper-hemispherical content into a representation that approximates an entire sphere.

3. The method for shared image processing of claim 1 , wherein the spherical content is obtained from a plurality of cameras.

4. The method for shared image processing of claim 1 , wherein:
the spherical content comprises a plurality of hyper-hemispherical content with at least one overlapping portion between the plurality of hyper-hemispherical content, and
the overlapping data structure comprises the at least one overlapping portion for each of the plurality of hyper-hemispherical content.

5. The method for shared image processing of claim 1 , wherein the overlapping data structure comprises an equi-angular cubemap spherical projection with retained overlap regions adjacent to each other at a stitch line in the equi-angular cubemap spherical projection.

6. The method for shared image processing of claim 1 , wherein the first portion of the multiband blend operation enables a device to display the overlapping data structure without performing an additional blending operation.

7. The method for shared image processing of claim 1 , wherein the first portion of the multiband blend operation enables a device to perform an additional blending operation on the overlapping data structure.

8. The method for shared image processing of claim 1 , wherein the first portion of the multiband blend operation comprises:
performing fewer scales of the multiband blend operation than a complete multiband blend operation;
creating low frequency metadata representing low frequency unblended portions of the spherical content; and
storing the low frequency metadata.

9. The method for shared image processing of claim 1 , further comprising enabling the second device to perform a second portion of the multiband blend operation on the first image processed data structure.

10. The method for shared image processing of claim 9 , further comprising displaying the first image processed data structure without performing an additional blending operation within a real-time budget.

11. The method of claim 1 , where converting the spherical content to the overlapping data structure comprises aligning overlapping regions of the spherical content.

12. An apparatus for shared image processing, comprising:
a first camera and a second camera;
a processor; and
a non-transitory computer-readable medium comprising one or more instructions which, when executed by the processor causes the apparatus to:
capture spherical content by the first camera and the second camera, the spherical content comprising overlapping regions captured by the first camera and the second camera;
align the spherical content to create an overlapping data structure;
perform a partial multiband blend on the overlapping data structure to create a partially processed overlapping data structure with low frequency metadata; and
transmit the partially processed overlapping data structure with the low frequency metadata to a second device.

13. The apparatus for shared image processing of claim 12 , further comprising a display, wherein the one or more instructions, when executed by the processor, further causes the apparatus to:
crop out portions of overlapping content from the partially processed overlapping data structure to create a cropped partially processed overlapping data structure; and
display the cropped partially processed overlapping data structure.

14. The apparatus for shared image processing of claim 12 , wherein the partial multiband blend is performed within idle periods of a real-time budget for capturing the spherical content by the first camera and the second camera.

15. The apparatus for shared image processing of claim 12 , wherein the partial multiband blend comprises performing fewer scales of a multiband blend operation than a complete multiband blend operation.

16. A method for multi-stage image processing, the method comprising:
receiving a data structure comprising image data of overlapping regions;
performing a first portion of a multi-band blend on the data structure based on the image data of overlapping regions;
saving low-frequency portions of the data structure that remain unblended after the first portion of the multi-band blend within the data structure creating a partially blended data structure with low-frequency metadata; and
transferring, by a first device, the partially blended data structure with low-frequency metadata to a second device configured to complete the multi-band blend.

17. The method of claim 16 , where the first portion of the multi-band blend comprises performing a partial multi-band blend of high-frequency portions of the data structure.

18. The method of claim 16 , further comprising:
receiving first image data from a first camera with a first hyper-hemispherical field-of-view lens;
receiving second image data from a second camera with a second hyper-hemispherical field-of-view lens; and
stitching the first image data and the second image data to create the data structure, where the data structure comprises an overlap band of image data from the first image data and the second image data, comprising at least a portion of the image data of overlapping regions.

19. The method of claim 16 , where the first portion of the multi-band blend comprises:
performing a number of downscale operations on the data structure; and
performing a different number of upscale operations on the data structure.

20. The method of claim 16 , where:
the first portion of the multi-band blend on the data structure is performed by the first device at a best effort, and
a number of scales of the first portion of the multi-band blend to perform is based on remaining resources available on the first device after prioritized tasks are allocated resources on the first device.

Assignees

• Gopro Inc

Inventors

• Marc Gladys Jocelyne
• Lefebvre Alexis
• Douady Cesar
• Guerin Guillaume

Classifications

• G06T3/12 Primary

  Panospheric to cylindrical image transformations · CPC title

• G06T3/16

  Spatio-temporal transformations, e.g. video cubism · CPC title

• G06T3/4038 Primary

  Image mosaicing, e.g. composing plane images from plane sub-images · CPC title

• G06T3/0087

  Physics · mapped topic

• G06T7/30

  Determination of transform parameters for the alignment of images, i.e. image registration · CPC title