Method, system and computer program product for automatically altering a video stream

Patent metadata

Field	Value
Publication number	US-2017337692-A1
Application number	US-201715660252-A
Country	US
Kind code	A1
Filing date	Jul 26, 2017
Priority date	Jan 27, 2015
Publication date	Nov 23, 2017
Grant date	—

Abstract

There is provided a method for automatically altering a digital video stream including multiple video input frames, to automatically obtain output frames with a target composition, in which frame metadata relating to objects in one or more of the video input frames is analyzed on a frame-by-frame basis and used by a processor to automatically alter one or more output frames to be more similar to, or to match, the target composition, wherein cropping is performed in 3D. A related system and a related computer program product are also provided.

First claim

1 . A method for automatically cropping input video frames of a digital video stream to obtain output video frames, wherein obtaining the output video frames comprises:
analyzing on a frame-by-frame basis frame metadata relating to objects in one or more of the input video frames; and using of the frame metadata by a processor to automatically crop one or more of the input video frames based on a target frame composition, wherein the target frame composition defines at least one of a number, identity and/or position of objects for inclusion in at least one output video frame, wherein the cropping is based on the detection of objects which have associated metrics, the metrics having three-dimensional space parameters.

2 . The method according to claim 1 , wherein at least one of the three-dimensional space parameters relates to depth information corresponding to the objects in the one or more of the input video frames.

3 . The method according to claim 2 , wherein the depth information is depth sensor data.

4 . The method according to claim 1 , wherein the cropping is based on detection of a single target object.

5 . The method according to claim 1 , wherein the cropping is based on detection of a plurality of target objects.

6 . The method according to claim 1 , wherein the target frame composition is based on a determined association between at least two objects for inclusion in the at least one output video frame.

7 . The method according to claim 1 , wherein the cropping is based on a scene type metric, the scene type metric identifying a given input video frame as depicting a predefined category of scene.

8 . The method according to claim 7 , wherein the scene type metric is based on a predefined rule describing speed of at least one object in the given input video frame.

9 . The method according to claim 1 , comprising determining a bounding surface area in the one or more input video frames, wherein:
the bounding area of a given input video frame comprises all of the objects for inclusion in the at least one output frame; and the cropping is based on the bounding area.

10 . The method according to claim 9 , in which the bounding area is a regular polyhedron in three-dimensional space.

11 . The method according to claim 1 , further comprising:
receiving, from a user and via a user interface, a selection of a given object in one or more of the input video frames, and: responsive to the selection, setting the target frame composition to define said given object for inclusion in at least one output video frame.

12 . A system comprising a processor and storage, configured to implement a method for automatically cropping input video frames of a digital video stream to obtain output video frames, wherein obtaining the output video frames comprises:
analyzing on a frame-by-frame basis frame metadata relating to objects in one or more of the input video frames; and using of the frame metadata by a processor to automatically crop one or more of the input video frames based on a target frame composition, wherein the target frame composition defines at least one of a number, identity and/or position of objects for inclusion in at least one output video frame, wherein the cropping is based on the detection of objects which have associated metrics, the metrics having three-dimensional space parameters.

13 . A method for automatically cropping input video frames of a digital video stream, to obtain output video frames, wherein obtaining the output video frames comprises:
analyzing on a frame-by-frame basis frame metadata relating to objects in one or more of the input video input frames; and using of the frame metadata by a processor to automatically crop one or more output frames based on a target composition, wherein the target composition is defined by:
a first set of rules that are functional rules, such as maximally filling a frame with an object, or placing an object in the center of a given output frame; and
a second set of rules that are aesthetic rules defining a desired aesthetic appearance of a given output frame.

14 . The method according to claim 13 , wherein metadata relating to a given object describes proportions of that object.

15 . The method according to claim 13 , wherein the metadata relating to a given object describes motion of that object.

16 . The method according to claim 13 , wherein metadata relating to a given object describes a position of that object within an input video frame.

17 . The method according to claim 13 , wherein metadata relating to a given object defines proportions of that object based on proportions of a rectangle fitted around that object.

18 . The method according to claim 13 , wherein the method is performed in real time.

19 . The method according to claim 13 , wherein the method is performed as post-processing.

Assignees

• Apical Ltd

Inventors

• Romanenko Ilya
• Terekhov Vladislav

Classifications

• G06T11/00 Primary

  Two-dimensional [2D] image generation · CPC title

• G06T7/11 Primary

  Region-based segmentation · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• G06K9/00201

  Physics · mapped topic

• G06T2207/10021

  Stereoscopic video; Stereoscopic image sequence · CPC title