Scene and activity identification in video summary generation

Patent metadata

Field	Value
Publication number	US-11069380-B2
Application number	US-201916382600-A
Country	US
Kind code	B2
Filing date	Apr 12, 2019
Priority date	Jul 23, 2014
Publication date	Jul 20, 2021
Grant date	Jul 20, 2021

Abstract

Video and corresponding metadata is accessed. Events of interest within the video are identified based on the corresponding metadata, and best scenes are identified based on the identified events of interest. A video summary can be generated including one or more of the identified best scenes. The video summary can be generated using a video summary template with slots corresponding to video clips selected from among sets of candidate video clips. Best scenes can also be identified by receiving an indication of an event of interest within video from a user during the capture of the video. Metadata patterns representing activities identified within video clips can be identified within other videos, which can subsequently be associated with the identified activities.

First claim

What is claimed is:

1. A system configured to generate a video summary, the system comprising:
storage media that stores a video of an activity including frames and audio captured contemporaneously with the frames;
one or more physical processors configured by machine readable instructions to:
obtain metadata for the video, the metadata including biometric information of a user captured contemporaneously with capture of the video and camera motion information characterizing position and/or motion of the camera during capture of the video;
analyze the biometric information, the camera motion information, and/or the audio to identify events of interest and types of the events of interest within the video;
select portions of the video for inclusion in the video summary, the portions including the events of interest, wherein different numbers of the frames are included within the portions based on different combinations of a type of the activity and the types of the events of interest such that the selected portions of the video include a first portion, the first portion including a first event of interest, and different numbers of the frames are included within the first portion based on whether the activity is a first activity type and the first event of interest is a first event type, the activity is the first activity type and the first event of interest is a second event type, the activity is a second activity type and the first event of interest is the first event type, or the activity is the second activity type and the first event of interest is the second event type; and
generate the video summary from the selected portions of the video;
wherein the selected portions of the video are arranged within the video summary based on a video summary template, the video summary template selected based on the type of the activity, the video summary template defining a pre-defined ordering of video portions for inclusion in the video summary based on the types of the events of interest.

2. The system of claim 1 , wherein the one or more physical processors are further configured to rank the events of interest within the video based on the biometric information, the camera motion information, and/or the audio.

3. The system of claim 2 , wherein the one or more physical processors are further configured such that selection of the portions of the video for inclusion in the video summary is further based on the rankings of the individual events of interest.

4. The system of claim 1 , wherein the one or more physical processors are further configured to provide individual selected portions of the video.

5. The system of claim 1 , wherein the events of interest include the first event of interest identified on the basis of a change in the position and/or motion of the camera during capture of the video during capture of the first event of interest as characterized by the camera motion information.

6. The system of claim 5 , wherein the one or more physical processors are configured such that identification of the first event of interest is based on the camera motion information for the first event of interest indicating a change in the position and/or motion of the camera.

7. The system of claim 1 , wherein the events of interest include the first event of interest identified on the basis of the audio captured contemporaneously with the first event of interest.

8. The system of claim 7 wherein one or more physical processors are configured such that identification of the first event of interest is based on one or more of a level change of the audio captured contemporaneously with the first event of interest, recognition of a word or phrase in the audio captured contemporaneously with the first event of interest, or a non-verbal pattern or noise present in the audio captured contemporaneously with the first event of interest.

9. The system of claim 1 , wherein the one or more physical processors are configured such that number of separate portions of the video selected for inclusion in the video summary is based on a user-selected summary length.

10. A method for generating a video summary, the method performed by a computing system including one or more processors and storage media that stores a video of an activity including frames and audio captured contemporaneously with the frames, the method comprising:
obtaining, by the computing system, metadata for the video, the metadata including biometric information of a user captured contemporaneously with capture of the video and camera motion information characterizing position and/or motion of the camera during capture of the video;
analyzing, by the computing system, the biometric information, the camera motion information, and/or the audio to identify events of interest and types of the events of interest within the video;
selecting, by the computing system, portions of the video for inclusion in the video summary, the portions including the events of interest, wherein different numbers of the frames are included within the portions based on different combinations of a type of the activity and the types of the events of interest such that the selected portions of the video include a first portion, the first portion including a first event of interest, and different numbers of the frames are included within the first portion based on whether the activity is a first activity type and the first event of interest is a first event type, the activity is the first activity type and the first event of interest is a second event type, the activity is a second activity type and the first event of interest is the first event type, or the activity is the second activity type and the first event of interest is the second event type; and
generating, by the computing system, the video summary from the selected portions of the video;
wherein the selected portions of the video are arranged within the video summary based on a video summary template, the video summary template selected based on the type of the activity, the video summary template defining a pre-defined ordering of video portions for inclusion in the video summary based on the types of the events of interest.

11. The method of claim 10 , further comprising ranking the events of interest within the video based on the biometric information, the camera motion information, and/or the audio.

12. The method of claim 11 , wherein selection of the portions of the video for inclusion in the video summary is further based on the rankings of the individual events of interest.

13. The method of claim 10 , further comprising providing individual selected portions of the video.

14. The method of claim 10 , wherein the events of interest include the first event of interest identified on the basis of a change in the position and/or motion of the camera during capture of the video during capture of the first event of interest as characterized by the camera motion information.

15. The method of claim 14 , wherein identification of the first event of interest is based on the camera motion information for the first event of interest indicating a change in the position and/or motion of the camera.

16. The method of claim 10 , wherein the events of interest include the first event of interest identified on the basis of the audio captured contemporaneously with the first event of interest.

17. The method of claim 16 wherein identification of the first event of interest is based on one or more of a level change of the audio captured contemporaneously with the first event of interest, recognition of a word or phrase

Assignees

• Gopro Inc

Inventors

• Newman David A
• Silver Adam

Classifications

• G06V20/47 Primary

  Detecting features for summarising video content · CPC title

• G06V20/41

  Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49) · CPC title

• G06V20/44

  Event detection · CPC title

• H04N5/77

  between a recording apparatus and a television camera · CPC title

• G10L2015/223

  Execution procedure of a spoken command · CPC title