Generating semantically meaningful video loops in a cinemagraph

Patent metadata

Field	Value
Publication number	US-9779774-B1
Application number	US-201615264789-A
Country	US
Kind code	B1
Filing date	Sep 14, 2016
Priority date	Jul 22, 2016
Publication date	Oct 3, 2017
Grant date	Oct 3, 2017

Abstract

A cinemagraph is generated that includes one or more video loops. A cinemagraph generator receives an input video, and semantically segments the frames to identify regions that correspond to semantic objects and the semantic object depicted in each identified region. Input time intervals are then computed for the pixels of the frames of the input video. An input time interval for a particular pixel includes a per-pixel loop period and a per-pixel start time of a loop at the particular pixel. In addition, the input time interval of a pixel is based, in part, on one or more semantic terms which keep pixels associated with the same semantic object in the same video loop. A cinemagraph is then created using the input time intervals computed for the pixels of the frames of the input video.

First claim

Wherefore, what is claimed is:

1. A system for generating a cinemagraph comprising one or more video loops, comprising:
a cinemagraph generator comprising one or more computing devices, said computing devices being in communication with each other via a computer network whenever there is a plurality of computing devices, and a computer program having a plurality of sub-programs executable by said computing devices, wherein the sub-programs configure said computing devices to,
receive an input video, wherein the input video comprises a sequence of frames each frame of which comprises pixels,
semantically segment the frames of the input video to identify regions in the frames that correspond to semantic objects,
compute input time intervals for the pixels of the frames of the input video, wherein an input time interval for a particular pixel comprises a per-pixel loop period and a per-pixel start time of a loop at the particular pixel, and wherein the input time interval of a pixel is based, in part, on semantic terms which keep pixels associated with the same semantic object in the same video loop, and
create a cinemagraph using the input time intervals computed for the pixels of the frames of the input video, wherein the cinemagraph exhibits regions that appear static to a viewer and regions comprising dynamic video loops that appear to the viewer to be changing over time.

2. The system of claim 1 , wherein the sub-program for computing input time intervals for the pixels of the frames of the input video, comprises determining the respective input time intervals for the pixels that optimize an objective function, wherein the objective function comprises a temporal consistency term which is indicative of a temporal semantic consistency and a temporal photometric consistency across the frames from a loop start frame to an end loop frame.

3. The system of claim 1 , wherein the sub-program for computing input time intervals for the pixels of the frames of the input video, comprises determining the respective input time intervals for the pixels that optimize an objective function, wherein the objective function comprises a static term which assigns a penalty value to static pixels.

4. The system of claim 1 , wherein the sub-program for computing input time intervals for the pixels of the frames of the input video, comprises determining the respective input time intervals for the pixels that optimize an objective function, wherein the objective function comprises a spatial consistency term which is indicative of a spatial semantic compatibility and a spatial photometric compatibility between spatially adjacent pixels over a loop period.

5. The system of claim 1 , wherein the semantic terms measure the semantic difference between pixels.

6. The system of claim 5 , wherein each semantic term is in the form a feature vector that represents the semantic difference between pixels based on at least one of a semantic label map, or semantic segmentation responses, or intermediate activation of a Convolutional Neural Network.

7. The system of claim 6 , wherein a principal component analysis is applied to the feature vector representing the semantic term to project the feature from a higher-dimensional vector space to a lower-dimensional vector space.

8. The system of claim 5 , wherein each semantic term is computed using two or more semantic segmentation methods, and the responses obtained from each of the semantic segmentation methods employed are aggregated.

9. A system for generating a cinemagraph comprising one or more video loops, comprising:
a cinemagraph generator comprising one or more computing devices, said computing devices being in communication with each other via a computer network whenever there is a plurality of computing devices, and a computer program having a plurality of sub-programs executable by said computing devices, wherein the sub-programs configure said computing devices to,
receive an input video, wherein the input video comprises a sequence of frames each frame of which comprises pixels,
semantically segment the frames of the input video to identify regions in the frames that correspond to semantic objects and the semantic object depicted in each identified region,
compute input time intervals for the pixels of the frames of the input video, wherein an input time interval for a particular pixel comprises a per-pixel loop period and a per-pixel start time of a loop at the particular pixel, and wherein the input time interval for pixels residing in regions corresponding to a semantic object is based, in part, on a semantic term which keep pixels associated with the same semantic object in the same video loop, and
create a cinemagraph using the input time intervals computed for the pixels of the frames of the input video, wherein the cinemagraph exhibits regions that appear static to a viewer and regions comprising dynamic video loops that appear to the viewer to be changing over time.

10. The system of claim 9 , wherein the sub-program for computing input time intervals for the pixels of the frames of the input video, comprises determining the respective input time intervals for the pixels that optimize an objective function, wherein the objective function comprises a temporal consistency term which is indicative of a temporal photometric consistency across the frames from a loop start frame to an end loop frame.

11. The system of claim 9 , wherein the sub-program for computing input time intervals for the pixels of the frames of the input video, comprises determining the respective input time intervals for the pixels that optimize an objective function, wherein the objective function comprises a static term which assigns a penalty value to static pixels.

12. The system of claim 9 , wherein the sub-program for computing input time intervals for the pixels of the frames of the input video, comprises determining the respective input time intervals for the pixels that optimize an objective function, wherein the objective function comprises either a spatial consistency term which is indicative of a spatial photometric compatibility between spatially adjacent pixels over a loop period, or the semantic term, whichever is greater for the pixel.

13. The system of claim 12 , wherein the sub-program for computing input time intervals for the pixels of the frames of the input video, comprises a sub-program for computing a semantic term for the pixels associated with each identified semantic object, wherein the semantic term computation comprises:
for each sequential pair of frames,
computing the ratio of the intersection of the regions in the pair of frames associated with the semantic object under consideration,
establishing a multiplier using the computed ratio that increases as the computed ratio increases,
computing the semantic term for the pair of frames under consideration as the product of the multiplier and a prescribed minimum semantic cost; and

establishing the semantic term for the pixels associated with the identified semantic object as the maximum semantic term computed among the pairs of frames.

14. The system of claim 13 , wherein computing the ratio of the intersection of the regions in a pair of frames associated with a semantic object comprises computing the ratio of the number of pixels associated with intersection of the regions in the pair of frames associated with the semantic object, and the number of pixels associated with the larger of the two regions in the pair of frames associated with the semantic object.

15. The system of claim 9 , wherein the sub-program for semantically segmenti

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Kang Sing Bing
• Joshi Neel Suresh
• Hoppe Hugues
• Oh Tae-Hyun
• Wang Baoyuan

Classifications

• G06V30/19173

  Classification techniques · CPC title

• G06V10/82

  using neural networks · CPC title

• G06F18/24133

  Distances to prototypes · CPC title

• G06F18/23

  Clustering techniques · CPC title

• G06V10/454

  Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN] · CPC title