Panoramic image stitching using objects

Patent metadata

Field	Value
Publication number	US-9911213-B2
Application number	US-201514975635-A
Country	US
Kind code	B2
Filing date	Dec 18, 2015
Priority date	Dec 18, 2015
Publication date	Mar 6, 2018
Grant date	Mar 6, 2018

Abstract

A system and method that determines a seam between pairs of adjacent images for panoramic image stitching is disclosed. The method includes receiving a sequence of images, determining a pair of adjacent images in the sequence of images, matching one or more objects corresponding to a same object identifier in the pair of adjacent images, determining a seam in an overlap region between the pair of adjacent images and determining a portion of pixels from each image of the pair of adjacent images to represent in a stitched panoramic image based on the seam.

First claim

What is claimed is:

1. A method comprising:
receiving, using one or more processors, a sequence of images of a scene;
determining, using the one or more processors, a pair of adjacent images in the sequence of images;
identifying, using the one or more processors, one or more objects corresponding to a same object identifier in the pair of adjacent images;
determining, using the one or more processors, a first set of corresponding features for each identified object in the pair of adjacent images;
determining, using the one or more processors, a homography for each identified object based on the first set of the corresponding features for each identified object;
adding, using the one or more processors, the homography for each identified object to a list;
clustering, using the one or more processors, homographies in the list;
refining, using the one or more processors, homographies of a best cluster; and
determining, using the one or more processors, a relationship between the pair of adjacent images in the sequence of images and the one or more objects in the pair of adjacent images based on the refined homographies of the best cluster.

2. The method of claim 1 , wherein refining the homographies of the best cluster comprises:
identifying a cluster with a largest number of homographies as the best cluster;
determining a second set of corresponding features associated with homographies included in the best cluster;
identifying inliers based on the second set of corresponding features consistently matching in the pair of adjacent images; and
identifying the pair of adjacent images as a pair of matching images based on the inliers.

3. The method of claim 1 , wherein determining the relationship between the pair of adjacent images in the sequence of images and the one or more objects in the pair of adjacent images comprises:
determining a relationship between the one or more objects in an overlap region between the pair of adjacent images;
determining a seam in the overlap region between the pair of adjacent images using the relationship between the one or more objects in the overlap region; and
determining a portion of pixels from each image of the pair of adjacent images to represent in a stitched panoramic image based on the seam.

4. The method of claim 3 , wherein determining the seam in the overlap region between the pair of adjacent images comprises:
identifying the overlap region between the pair of adjacent images;
identifying a plurality of pixel locations within the overlap region for each adjacent image of the pair;
for each adjacent image of the pair, setting a value of an object indicator for the plurality of pixel locations based on whether an object is present at the plurality of pixel locations;
for each adjacent image of the pair, identifying a color value at the plurality of pixel locations;
computing a cost between each pair of adjacent pixel locations in the plurality of pixel locations within the overlap region as a weighted sum of the object indicator and color similarity; and
determining the seam that runs between one or more pairs of adjacent pixel locations in the plurality of pixel locations within the overlap region such that a total cost is minimized.

5. The method of claim 3 , wherein determining the seam in the overlap region between the pair of adjacent images comprises:
identifying the overlap region between the pair of adjacent images;
for each adjacent image of the pair, identifying the one or more objects within the overlap region, selecting a distinct color value, and modifying color values for a set of pixel locations forming a boundary around the one or more objects to the distinct color value;
for each adjacent image of the pair, identifying a color value at each pixel location within the overlap region;
computing a cost between each pair of adjacent pixel locations in the overlap region as a measure of color similarity; and
determining the seam that runs between pairs of adjacent pixel locations in the overlap region such that a total cost is minimized.

6. The method of claim 3 , comprising combining seams for all pairs of adjacent images in the stitched panoramic image.

7. The method of claim 3 , wherein the seam is placed in between the one or more objects in the overlap region between the pair of adjacent images.

8. The method of claim 1 , wherein determining the relationship between the pair of adjacent images in the sequence of images and the one or more objects in the pair of adjacent images includes performing bundle adjustment based on a pattern of image capture performed by a capture device.

9. The method of claim 8 , wherein the pattern of image capture is based on a grid.

10. The method of claim 8 , further comprising:
creating a fake image that represents the pattern of image capture; and
performing the bundle adjustment on the images using features from the sequence of images and the fake image.

11. A system comprising:
one or more processors; and
a memory, the memory storing instructions, which when executed cause the one or more processors to:
receive a sequence of images of a scene;
determine a pair of adjacent images in the sequence of images;
identify one or more objects corresponding to a same object identifier in the pair of adjacent images;
determine a first set of corresponding features for each identified object in the pair of adjacent images;
determine a homography for each identified object based on the first set of corresponding features for each identified object;
add the homography for each identified object to a list;
cluster homographies in the list;
refine homographies of a best cluster; and
determine a relationship between the pair of adjacent images in the sequence of images and the one or more objects in the pair of adjacent images based on the refined homographies of the best cluster.

12. The system of claim 11 , wherein to refine the homographies of the best cluster, the instructions cause the one or more processors to:
identify a cluster with a largest number of homographies as the best cluster;
determine a second set of corresponding features associated with homographies included in the best cluster;
identify inliers based on the second set of corresponding features consistently matching in the pair of adjacent images; and
identify the pair of adjacent images as a pair of matching images based on the inliers.

13. The system of claim 11 , wherein to determine the relationship between the pair of adjacent images in the sequence of images and the one or more objects in the pair of adjacent images, the instructions cause the one or more processors to:
determine a relationship between the one or more objects in an overlap region between the pair of adjacent images;
determine a seam in the overlap region between the pair of adjacent images using the relationship between the one or more objects in the overlap region; and
determine a portion of pixels from each image of the pair of adjacent images to represent in a stitched panoramic image based on the seam.

14. The system of claim 13 , wherein to determine the seam in the overlap region between the pair of adjacent images, the instructions cause the one or more processors to:
identify the overlap region between the pair of adjacent images;
for each adjacent image of the pair, identify the one or more objects within the overlap region, selecting a distinct color value and modifying color values for a set of pixel locations forming a boundary around the one or more objects to the distinct color value;
for each adjacent image of the pair, identify

Assignees

• Gormish Michael
• Schwartz Edward
• Ricoh Co Ltd

Inventors

• Gormish Michael
• Schwartz Edward

Classifications

• G06T7/33 Primary

  using feature-based methods · CPC title

• G06V10/762

  using clustering, e.g. of similar faces in social networks · CPC title

• G06T11/60 Primary

  Creating or editing images; Combining images with text · CPC title

• G06F18/22

  Matching criteria, e.g. proximity measures · CPC title

• G06F18/23

  Clustering techniques · CPC title