Automatic video montage generation

Patent metadata

Field	Value
Publication number	US-11468915-B2
Application number	US-202017060878-A
Country	US
Kind code	B2
Filing date	Oct 1, 2020
Priority date	Oct 1, 2020
Publication date	Oct 11, 2022
Grant date	Oct 11, 2022

Abstract

In various examples, users may access a tool that automatically generates video montages from video clips of the user's gameplay according to parameterized recipes. As a result, a user may select—or allow the system to select—clips corresponding to gameplay of the user and customize one or more parameters (e.g., transitions, music, audio, graphics, etc.) of a recipe, and a video montage may be generated automatically according to a montage script output using the recipe. As such, a user may have a video montage generated with little user involvement, and without requiring any skill or expertise in video editing software. In addition, even for experienced video editors, automatic video montage generation may be a useful alternative to save the time and effort of manually curating video montages.

First claim

What is claimed is:

1. A method comprising:
instantiating a graphics generation engine using an executable program of an embedded browser client;
generating, using the graphics generation engine and based at least on a montage recipe, a video montage script defining a timeline fora video montage that includes at least a first video clip, a second video clip, and one or more graphical elements; and
generating the video montage, the generating including, for at least one montage frame of the video montage:
generating, using the graphics generation engine and based at least in part on the montage recipe, at least one graphical overlay frame including at least one graphical element of the one or more graphical elements and corresponding to a time on the timeline;
retrieving, using a transcoder and according to the video montage script, a video frame from the first video clip corresponding to the time, the transcoder being instantiated using the graphics generation engine; and
compositing, using the transcoder and according to the video montage script, the at least one graphical overlay frame and the video frame to generate the at least one montage frame corresponding to the time.

2. The method of claim 1 , wherein, for any montage frames of the at least one montage frame that have no corresponding video frames from the first video clip or the second video clip, the montage frames correspond to the at least one graphical overlay frame generated by the graphics generation engine.

3. The method of claim 1 , wherein, for any montage frames of the video montage that have no corresponding graphical overlay frame or a fully transparent graphical overlay frame, the montage frames correspond to the video clip frames of at least one of the first video clip or the second video clip.

4. The method of claim 1 , further comprising: compositing, using the transcoder, another video frame with the video frame as a picture-in-picture to generate a picture-in-picture composite frame, wherein the compositing the at least one graphical overlay frame and the video frame includes compositing the at least one graphical overlay frame and the picture-in-picture composite frame.

5. The method of claim 1 , wherein each video frame of the video montage is generated using a corresponding graphical overlay frame, the compositing is in red green blue alpha (RGBA) format, and the frames of the video montage that do not include a corresponding graphical element include the corresponding graphical overlay frames with fully transparent alpha values.

6. The method of claim 1 , wherein the retrieving the video frame is based at least in part on a command from the graphics generation engine indicating that the graphical overlay frame has been rendered.

7. The method of claim 1 , wherein the graphics generation engine is executed using an embedded browser and the graphics generation engine corresponds to a two-dimensional (2D) vector graphics generation engine.

8. The method of claim 1 , wherein a rendering dock of the graphics generation engine is advanced at a rate corresponding to capabilities of the transcoder.

9. The method of claim 1 , wherein the montage recipe is represented using code, the graphics generation engine executes a first pass of the code using a hypertext markup language (HTML) application to generate the video montage script, and the graphics generation engine executes a second pass of the code using the HTML application to generate the at least one graphical overlay frame.

10. The method of claim 1 , wherein the compositing is based at least in part on the transcoder selecting the at least one graphical overlay frame and the video frame according to the timeline.

11. The method of claim 1 , further comprising determining the montage recipe based at least in part on a game application or a type of game application that the first video clip and the second video clip correspond to.

12. The method of claim 1 , further comprising:
receiving input data representative of one or more selections of one or more preferences corresponding to the video montage; and
updating one or more parameters of the montage recipe based at least in part on the one or more preferences.

13. The method of claim 1 , wherein the first video clip and the second video clip correspond to segments of a single video.

14. The method of claim 1 , further comprising:
mining event logs generated based at least in part on an analysis of application data corresponding to one or more sessions of an application;
determining a first event and a second event based at least in part on the mining; and
retrieving or generating the first video clip corresponding to the first event and the second video clip corresponding to the second event,
wherein the receiving the video data is based at least in part on the retrieving or the generating the first video clip corresponding to the first event and the second video clip corresponding to the second event.

15. A system comprising:
one or more processors;
one or more memory devices storing instructions that, when executed using the one or more processors, cause the one or more processors to execute operations comprising:
receiving data representative of one or more user preferences for generation of a video montage;
based at least in part on the one or more user preferences, adjusting one or more parameters of a template montage recipe to generate a final montage recipe;
instantiating a graphics generation engine using an executable program of an embedded browser client;
generating, using the graphics generation engine and using the final montage recipe, a video montage script according to the final montage recipe, the video montage script defining a timeline for the video montage that identifies first locations along the timeline for a plurality of video clips and second locations along the timeline for one or more graphical overlays;
rendering, using the graphics generation engine and using the final montage recipe, one or more first frames corresponding to the one or more graphical overlays;
retrieving, using a transcoder and according to the video montage script, one or more second frames corresponding to the plurality of video clips, the transcoder being instantiated using the graphics generation engine; and
generating, using the transcoder and according to the video montage script, the video montage such that, for a first frame of the one or more first frames and a second frame of the one or more second frames that correspond to a same time on the timeline, a composite frame is generated from the first frame and the second frame.

16. The system of claim 15 , wherein the user preferences correspond to at least one of a selection of at least one of the plurality of video clips, an indication of one or more segments of a larger video that correspond to at least one of the plurality of video clips, a selection of at least one transition type between video clips of the plurality of video clips, an audio selection, a graphics selection, a text selection, an effect selection, or a theme selection for the video montage.

17. The system of claim 15 , wherein the operations further comprise:
generating a visual preview of the video montage;
receiving input data representative of one or more adjustments to the video preview; and
updating the final montage recipe based at least in part on the one or more adjustments.

18. The system of claim 15 , wherein the retrieving the one or more second frames includes, for each second frame of the one or more second frames, re

Assignees

• Nvidia Corp

Inventors

• Van Welzen James
• White Jonathan
• Clark David
• Cochran Glenn
• Parikh Amit

Classifications

• G11B27/036 Primary

  Insert-editing · CPC title

• G06V20/44

  Event detection · CPC title

• G06V20/41

  Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49) · CPC title

• G11B27/031 Primary

  Electronic editing of digitised analogue information signals, e.g. audio or video signals · CPC title

• G06T11/60

  Creating or editing images; Combining images with text · CPC title