Multiple camera video image stitching by placing seams for scene objects

Patent metadata

Field	Value
Publication number	US-9842624-B2
Application number	US-201514939865-A
Country	US
Kind code	B2
Filing date	Nov 12, 2015
Priority date	Nov 12, 2015
Publication date	Dec 12, 2017
Grant date	Dec 12, 2017

Abstract

Image stitching is described for multiple camera video by placing seams for objects in the scene. An object of interest is identified within an overlap between two images each from a different adjacent camera of a multiple camera system. An identified object of interest is placed within an identified one of the two images. A seam is placed between the two images so that the object of interest is within the identified image and not within the other of the two images. The two images are joined at the placed seam and the two joined images are rendered as a single image joined at the placed seam.

First claim

What is claimed is:

1. A method comprising:
identifying regions of of interest within an overlap between two images each from a different adjacent camera of a multiple camera system, the overlap being an image region in which an edge of a field of view of one of the two cameras overlaps an adjacent edge of a field of view of the other of the two cameras so that the overlap is in both of the two images;
determining that the identified regions of interest are within an identified one of the two images;
assigning a weight to each identified region of interest;
determining a plurality of costs, the costs being based on a combination of the weight and a displacement of the corresponding seam required to move the corresponding seam from the respective identified region of interest;
minimizing a cost function based on the determined costs to select a seam placement:
placing a seam between the two images so that the selected region of interest is within the identified image and not within the other of the two images by rotating a camera support that carries the different adjacent cameras of the multiple camera system;
joining the two images at the placed seam; and
rendering the two images as a single image joined at the placed seam.

2. The method of claim 1 , wherein placing the seam comprises searching for a seam position and confining the search to the overlap.

3. The method of claim 1 , wherein identifying regions of interest comprises identifying one of a face using a face identification module; text using a text identification module, or a display screen using a display screen detector module.

4. The method of claim 1 , wherein identifying regions of interest comprises tracking a location of a previously identified object of interest.

5. The method of claim 1 , wherein placing a seam comprises:
drawing a seam at an initial position;
determining whether the drawn seam passes through the the selected region of interest; and
using a boundary of the selected region of interest to move the seam to avoid the boundary.

6. The method of claim 1 , wherein the boundary is used as a constraint in a seam placement process.

7. The method of claim 6 , further comprising selecting a second region of interest within the overlap and using a boundary of the second region of interest as a second constraint in minimizing the cost function.

8. The method of claim 1 , wherein determining that the identified-regions of interest are within an identified one of the two images comprises determining that the identified-regions of interest are entirely within an identified one of the two images.

9. A method to minimize or reduce the regions of interest upon which a seam fails, the method comprising:
identifying regions of interest that occur near scams between images each from a different adjacent camera of a multiple camera system;
assigning a weight to each identified region of interest;
determining a plurality of costs, the costs being based on a combination of the weight and a displacement of the corresponding seam required move the corresponding seam from the respective identified region of interest;
minimizing a cost function based on the determined costs to select a seam placement;
placing seams between the two images using the minimized cost;
joining the images at the placed seams; and
rendering the images as a single image joined at the placed seams.

10. The method of claim 9 , wherein the weights are assigned based on the importance to a viewer of the respective region of interest.

11. The method of claim 9 , wherein the weights are assigned based on a category of an object in each respective region of interest.

12. The method of claim 9 , wherein the cost function is a gradient descent.

13. The method of claim 9 , wherein minimizing the cost function comprises selecting a series of displacement values and determining a cost until the cost converges.

14. The method of claim 9 , wherein placing seams comprises rotating a camera support that carries the different adjacent cameras of the multiple camera system.

15. A video conference node comprising:
a system processor;
a communication interface coupled to the system processor; and
a multiple camera system coupled to the system processor, the multiple camera system capturing at least two images each from a different adjacent camera, the images having an overlap being an image region in which an edge of a field of view of one of the two cameras overlaps an adjacent edge of a field of view of the other of the two cameras so that the overlap is in both of the two images, the system processor identifying an object of interest within the overlap, determining that the identified object of interest is within an identified one of the two images, placing a seam between the two images so that the object of interest is within the identified image and not within the other of the two images by rotating a camera support that carries the different adjacent cameras of the multiple camera system, joining the two images at the placed seam, and rendering the two images as a single image joined at the placed seam,
wherein the system processor places the seam by using a cost function, the cost function including identifiying regions of interest that occur near multiple seam between multiple images each from a different adjacent camera of the multiple camera system, assigning a weight to each identified region of interest, and minimizing a cost fuction based on a combination of the weight and a displacement of a corresponding seam.

16. The video conference node of claim 15 , wherein the system processor comprises a central processing unit and an imaging chip, the imaging chip joining the two images and rendering the two images.

17. The video conference node of claim 15 , wherein placing the seam comprises:
drawing a seam at an initial position;
determining whether the drawn seam passes through the object of interest; and
using a bindary of the object of interest to move the seam to avoid the boundary.

18. The video conference node of claim 17 , wherein the boundary is used as a constraint in a seam placement process, the system processor further identifying a second object of interest within the overlap and using a boundary of the second object of interest as a second constraint in the seam placement process.

19. The video conference node of claim 15 , wherein the multiple camera system comprises a plurality of cameras mounted to a central support, equally spaced around the perimeter of the support and directed outwards from the center of the support to provide a panoramic field of view.

Assignees

• Intel Corp

Inventors

• Das Barnan
• Baran Stanley
• Hicks Richmond
• PYO Sean
• Lee Kanghee

Classifications

• G11B27/036 Primary

  Insert-editing · CPC title

• G06T11/60 Primary

  Creating or editing images; Combining images with text · CPC title

• H04N23/698

  for achieving an enlarged field of view, e.g. panoramic image capture · CPC title

• H04N23/58

  Means for changing the camera field of view without moving the camera body, e.g. nutating or panning of optics or image sensors · CPC title

• H04N23/90

  Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums · CPC title