Apparatus and methods for the storage of overlapping regions of imaging data for the generation of optimized stitched images

Patent metadata

Field	Value
Publication number	US-11477395-B2
Application number	US-202117240868-A
Country	US
Kind code	B2
Filing date	Apr 26, 2021
Priority date	Jan 13, 2017
Publication date	Oct 18, 2022
Grant date	Oct 18, 2022

Abstract

Apparatus and methods for stitching images, or re-stitching previously stitched images. Specifically, the disclosed systems in one implementation save stitching information and/or original overlap source data during an original stitching process. During subsequent retrieval, rendering, and/or display of the stitched images, the originally stitched image can be flexibly augmented, and/or re-stitched to improve the original stitch quality. Practical applications of the disclosed solutions enable, among other things, a user to create and stitch a wide field of view (FOV) panorama from multiple source images on a device with limited processing capability (such as a mobile phone or other capture device). Moreover, post-processing stitching allows for the user to convert from one image projection to another without fidelity loss (or with an acceptable level of loss).

First claim

What is claimed:

1. An apparatus, the apparatus comprising:
a graphical user interface configured to obtain user instructions;
a processor; and
a non-transitory computer-readable apparatus in data communication with the processor, the non-transitory computer-readable apparatus comprising a plurality of instructions that, when executed by the processor, cause the apparatus to:
obtain a stitched image having a first stitch line;
determine a first area of the stitched image to re-stitch with a first fidelity and a first perspective based on a first user instruction;
determine a second area of the stitched image that may be distorted during the re-stitch;
obtain a first overlap portion and a second overlap portion corresponding to the first area; and
re-stitch the first area of the stitched image based on the first overlap portion and the second overlap portion.

2. The apparatus of claim 1 , where the plurality of instructions, when executed by the processor, further cause the apparatus to determine a second area of the stitched image that may be passed-through the re-stitch.

3. The apparatus of claim 1 , where the first user instruction comprises reconciliation of an image object from the first overlap portion or the second overlap portion for inclusion within the re-stitch of the first area.

4. The apparatus of claim 1 , where the first user instruction merges a first image object from the first overlap portion with a ghost image object from the second overlap portion for the re-stitch of the first area.

5. The apparatus of claim 1 , where the stitched image is part of a video, and where the first user instruction identifies temporal portions of the video to be smoothed, blurred, or sharpened.

6. The apparatus of claim 1 , where:
the stitched image is part of a video, and
the plurality of instructions, when executed by the processor, further cause the apparatus to selectively stitch a portion of the video to allow a user to preview the re-stitch.

7. A method of re-stitching an image based on user instruction, the method comprising:
obtaining a stitched image having a stitch line, where the stitched image comprises a fold-in region that is not displayed in the stitched image;
obtaining a user instruction to shift the stitch line;
re-stitching the stitched image based on intervening data that is recovered from the fold-in region, where the stitched image is associated with a first stitching technique and re-stitching the stitched image is based on a second stitching technique; and
displaying a re-stitched image.

8. The method of claim 7 , where re-stitching the stitched image only comprises re-stitching a portion of the stitched image.

9. The method of claim 7 , where the stitched image is a frame of a video and where the re-stitching is performed on-the-fly during display.

10. The method of claim 9 , where at least one other frame of the video is passed through without re-stitching.

11. The method of claim 9 , where the stitched image is associated with a first perspective and where re-stitching the stitched image changes the first perspective.

12. The method of claim 7 , where the first stitching technique is a cut-and-feather stitch performed on a mobile device.

13. The method of claim 7 , where the second stitching technique shifts the stitch line to correct for parallax or a blind spot.

14. The method of claim 7 , where the second stitching technique comprises performing feature detection on an overlapping region of the stitched image recovered from the fold-in region.

15. An apparatus, comprising:
a first camera configured to capture a first image;
a second camera configured to capture a second image;
a graphical user interface configured to obtain user instructions;
a processor; and
a non-transitory computer-readable apparatus in data communication with the processor, the non-transitory computer-readable apparatus comprising a plurality of instructions that, when executed by the processor, cause the apparatus to:
capture the first image;
capture the second image;
stitch the first image and the second image to generate: a first overlapping portion, a second overlapping portion, a first non-overlapping portion, and a second non-overlapping portion; and
generate a stitched image data structure that tags the first non-overlapping portion and the second non-overlapping portion for display and tags the first overlapping portion and the second overlapping portion for re-stitching.

16. The apparatus of claim 15 , where the plurality of instructions, when executed by the processor, cause the apparatus to stitch based on a cut-and-feather operation over a first portion.

17. The apparatus of claim 16 , where the plurality of instructions, when executed by the processor, cause the apparatus to stitch based on a depth-based operation over a second portion.

18. The apparatus of claim 16 , where the first non-overlapping portion and the second non-overlapping portion are compressed in a panoramic image format; and
where the first overlapping portion and the second overlapping portion are compressed in a generic data compression format.

19. The apparatus of claim 16 , where the first non-overlapping portion and the first overlapping portion are associated with a first set of capture parameters of the first camera and where the second non-overlapping portion and the second overlapping portion are associated with a second set of capture parameters of the second camera.

20. The apparatus of claim 16 , where the stitched image data structure further comprises stitching metadata comprising at least a confidence metric.

Assignees

• Gopro Inc

Inventors

• Abbas Adeel
• Douady Cesar
• Macmillan Timothy

Classifications

• H04N23/698 Primary

  for achieving an enlarged field of view, e.g. panoramic image capture · CPC title

• H04N23/90

  Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums · CPC title

• H04N5/265 Primary

  Mixing · CPC title

• H04N7/181

  for receiving images from a plurality of remote sources · CPC title

• H04N5/247

  Electricity · mapped topic