Multi-view scene segmentation and propagation

Patent metadata

Field	Value
Publication number	US-10275892-B2
Application number	US-201715462752-A
Country	US
Kind code	B2
Filing date	Mar 17, 2017
Priority date	Jun 9, 2016
Publication date	Apr 30, 2019
Grant date	Apr 30, 2019

Abstract

A depth-based effect may be applied to a multi-view video stream to generate a modified multi-view video stream. User input may designate a boundary between a foreground region and a background region, at a different depth from the foreground region, of a reference image of the video stream. Based on the user input, a reference mask may be generated to indicate the foreground region and the background region. The reference mask may be used to generate one or more other masks that indicate the foreground and background regions for one or more different images, from different frames and/or different views from the reference image. The reference mask and other mask(s) may be used to apply the effect to the multi-view video stream to generate the modified multi-view video stream.

First claim

What is claimed is:

1. A method for applying an effect to a multi-view video stream, the method comprising:
in a data store, receiving the video stream;
in an input device, receiving first user input designating a boundary between a foreground region and a background region, at a different depth from the foreground region, of a reference image of the video stream;
in a processor, based on the first user input, generating a reference mask that indicates the foreground region and the background region;
in the processor, using the reference mask to generate a first non-reference mask that indicates the foreground region and the background region for a first non-reference image, different from the reference image, of the video stream;
in the processor, generating a modified video stream by using the reference mask to apply the effect to the reference image and using the first non-reference mask to apply the effect to the first non-reference image; and
in a display device, displaying the modified video stream as a virtual reality or augmented reality experience.

2. The method of claim 1 , further comprising, in a camera, capturing the video stream prior to receiving the video stream in the data store.

3. The method of claim 1 , wherein:
the reference image is from a reference frame of the video stream; and
the first non-reference image is from a first frame, different from the reference frame, of the video stream.

4. The method of claim 3 , further comprising, in the processor, using the first non-reference mask to generate a second non-reference mask that indicates the foreground region and the background region for a second non-reference image, from a second frame, different from the reference frame and the first frame, of the video stream;
wherein generating the modified video stream further comprises using the second non-reference mask to apply the effect to the second non-reference image.

5. The method of claim 1 , wherein:
the reference image is from a reference view of a reference frame of the video stream; and
the first non-reference image is from a first view, different from the reference view, of the reference frame of the video stream.

6. The method of claim 5 , further comprising, in the processor, using the first non-reference mask to generate a second non-reference mask that indicates the foreground region and the background region for a second non-reference image, from a second view, different from the reference view and the first view, of the reference frame of the video stream;
wherein generating the modified video stream further comprises using the second non-reference mask to apply the effect to the second non-reference image.

7. The method of claim 1 , further comprising, in the input device, prior to receiving the first user input, receiving second user input designating the reference image from among a plurality of images of the video stream.

8. The method of claim 1 , wherein generating the reference mask comprises:
using the boundary to compute an initial binary segmentation designating the foreground region and the background region of the reference image;
refining the boundary to delineate a refined boundary that more accurately designates the foreground region and the background region; and
using the refined boundary to generate the reference mask.

9. The method of claim 8 , wherein using the refined boundary to generate the reference mask comprises using the refined boundary to generate a trimap that further indicates an unknown region between the foreground region and the background region, the unknown region containing pixels that may belong to the foreground region or the background region.

10. The method of claim 9 , further comprising, in the processor, applying a matting algorithm to the trimap to obtain an alpha matte comprising alpha values for the pixels in the foreground region, the background region, and the unknown region.

11. The method of claim 10 , wherein the video stream comprises light-field video captured by one or more light-field cameras.

12. The method of claim 11 , further comprising back-projecting the alpha matte and a foreground image of the foreground region to the light-field video.

13. The method of claim 9 , wherein using the reference mask to generate the first non-reference mask comprises:
designating a first non-reference boundary between a first non-reference segment and a second non-reference segment of the first non-reference image;
using the first non-reference boundary to compute an initial non-reference binary segmentation designating the first non-reference segment and the second non-reference segment of the first non-reference image;
refining the first non-reference boundary to delineate a refined non-reference boundary that more accurately designates the first non-reference segment and the second non-reference segment; and
using the refined non-reference boundary to generate a non-reference trimap that further indicates an unknown non-reference segment between the first non-reference segment and the second non-reference segment, the unknown non-reference segment containing non-reference pixels that may belong to the first non-reference segment or the second non-reference segment.

14. The method of claim 13 , further comprising, in the processor, applying a matting algorithm to the non-reference trimap to obtain a non-reference alpha matte comprising non-reference alpha values for the non-reference pixels in the unknown non-reference segment, to generate a non-reference foreground image of the foreground region.

15. A non-transitory computer-readable medium for applying an effect to a multi-view video stream, comprising instructions stored thereon, that when executed by a processor, perform the steps of:
causing a data store to receive the video stream;
causing an input device to receive first user input designating a boundary between a foreground region and a background region, at a different depth from the foreground region, of a reference image of the video stream;
based on the first user input, generating a reference mask that indicates the foreground region and the background region;
using the reference mask to generate a first non-reference mask that indicates the foreground region and the background region for a first non-reference image, different from the reference image, of the video stream;
generating a modified video stream by using the reference mask to apply the effect to the reference image and using the first non-reference mask to apply the effect to the first non-reference image; and
causing a display device to display the modified video stream as a virtual reality or augmented reality experience.

16. The non-transitory computer-readable medium of claim 15 , further comprising instructions stored thereon, that when executed by a processor, cause a camera to capture the video stream prior to receiving the video stream in the data store.

17. The non-transitory computer-readable medium of claim 15 , wherein:
the reference image is from a reference frame of the video stream;
the first non-reference image is from a first frame, different from the reference frame, of the video stream;
the non-transitory computer-readable medium further comprises instructions stored thereon, that when executed by a processor, uses the first non-reference mask to generate a second non-reference mask that indicates the foreground region and the background region for a second non-reference image, from a second frame, different from the reference frame and the first frame, of the video stream

Assignees

• Google Llc

Inventors

• Bleibel Francois
• Du Tingfang
• Nonn Thomas
• Tan Jie

Classifications

• G06T2207/20092

  Interactive image processing based on input by user · CPC title

• H04N5/272 Primary

  Means for inserting a foreground image in a background image, i.e. inlay, outlay · CPC title

• G06T11/60

  Creating or editing images; Combining images with text · CPC title

• G06T2207/10024

  Color image · CPC title

• H04N5/2625

  for obtaining an image which is composed of images from a temporal image sequence, e.g. for a stroboscopic effect (sequence generated by event triggered capturing H04N7/188) · CPC title