Cataloging video and creating video summaries

Patent metadata

Field	Value
Publication number	US-9620168-B1
Application number	US-201514976192-A
Country	US
Kind code	B1
Filing date	Dec 21, 2015
Priority date	Dec 21, 2015
Publication date	Apr 11, 2017
Grant date	Apr 11, 2017

Abstract

A system and method for determining video clips including interesting content from video data. The system may receive annotation data identifying time and positions corresponding to objects represented in the video data and the system may determine priority metrics associated with each of the objects. By associating the priority metrics with the time and positions corresponding to the objects, the system may generate a priority metric map indicating a time and position of interesting moments in the video data. The system may generate moments and/or video clips based on the priority metric map. The system may determine a time (e.g., video frames) and/or space (e.g., pixel coordinates) associated with the moments/video clips and may simulate camera motion such as panning and/or zooming with the moments/video clips. The system may generate a Master Clip Table including the moments, video clips and/or annotation data associated with the moments/video clips.

First claim

What is claimed is:

1. A computer-implemented method for selecting a theme, the method comprising:
receiving panoramic video data comprising panoramic video frames having a first aspect ratio greater than 2:1;
receiving annotation data associated with the panoramic video data;
selecting, using the annotation data, a first face represented in the panoramic video data;
determining, using the annotation data, first pixel coordinates that corresponds to a position of the first face represented in a first video frame of the panoramic video data;
determining, using the annotation data, second pixel coordinates that corresponds to a position of the first face represented in a second video frame of the panoramic video data;
selecting, using the annotation data, a second face represented in the panoramic video data;
determining, using the annotation data, third pixel coordinates that corresponds to a position of the second face represented in the first video frame;
determining, using the annotation data, fourth pixel coordinates that corresponds to a position of the second face represented in the second video frame;
determining, using the annotation data, a first priority metric that corresponds to a first measure of interest associated with the first video frame, the determining further comprising:
determining that the first face is represented in the first video frame,
determining that the second face is represented in the first video frame,
determining that a distance between the first pixel coordinates and the third pixel coordinates is below a first threshold, and
determining a color histogram using the first video frame,
determining an image quality score using the color histogram, and
determining that the image quality score is above a second threshold;

determining that the first priority metric is above a third threshold;
determining, using the annotation data, a second priority metric that corresponds to a second measure of interest associated with the second video frame;
determining that the second priority metric is above the third threshold;
selecting first video frames that begin with the first video frame, end with the second video frame and include all video frames having a time stamp after a first time stamp of the first video frame and before a second time stamp of the second video frame;
determining a first region of interest in the first video frame that includes the first pixel coordinates and the third pixel coordinates;
determining a second region of interest in the second video frame that includes the second pixel coordinates and the fourth pixel coordinates; and
generating output video data that has a second aspect ratio less than 2:1 and includes portions of the first video frames including the first region of interest and the second region of interest.

2. The computer-implemented method of claim 1 , further comprising:
determining a first cropped window from the first video frame, the first cropped window comprising a portion of the first video frame including the first pixel coordinates and the third pixel coordinates, wherein the first cropped window:
has the second aspect ratio,
has a first size, and
is located at a first position within the first video frame;

determining a second cropped window from the second video frame, the second cropped window comprising a portion of the second video frame including the second pixel coordinates and the fourth pixel coordinates, wherein the second cropped window:
has the second aspect ratio,
has a second size different from the first size, and
is located at a second position within the second video frame; and

generating the output video data including the first cropped window and the second cropped window, wherein the output video data represents at least one of:
panning from the first cropped window to the second cropped window, or
zooming from the first cropped window to the second cropped window.

3. The computer-implemented method of claim 1 , further comprising:
determining, using the annotation data, a third priority metric that corresponds to a third measure of interest in a third video frame;
determining that the third priority metric is above the third threshold;
determining, using the annotation data, a fourth priority metric that corresponds to a fourth measure of interest in a fourth video frame;
determining that the fourth priority metric is above the third threshold;
selecting second video frames that begin with the third video frame, end with the fourth video frame and include all video frames having a time stamp after a third time stamp of the third video frame and a fourth time stamp of the fourth video frame;
determining, using the annotation data, a first similarity score that corresponds to a similarity between the first video frames and the second video frames;
determining that the first similarity score is below a fourth threshold; and

generating second output video data that has the second aspect ratio and includes the second video frames.

4. The computer-implemented method of claim 1 , further comprising:
determining, using the annotation data, an overall priority metric for the output video data;
determining, using the annotation data, a first similarity score that corresponds to a similarity between the first video frame and the second video frame;
identifying, using the first similarity score, a first transition point within the output video data;
determining a video fingerprint associated with the output video data that indicates characteristics of the output video data; and
updating a table to include entries associated with the output video data, the table indicating the first video frames, the overall priority metric, the first transition point, the video fingerprint, the first pixel coordinates associated with the first face in the first video frame and the second pixel coordinates associated with the first face in the second video frame.

5. A computer-implemented method, comprising:
receiving input video data that has a first aspect ratio greater than 2:1;
receiving annotation data associated with the input video data;
determining, using the annotation data, a plurality of pixel coordinates associated with a first object represented in the input video data;
determining, using the annotation data, a first priority metric that corresponds to a first measure of interest associated with a first video frame of the input video data;
determining that the first priority metric is above a threshold;
selecting the first video frame that corresponds to the first priority metric;
determining a first region of interest in the first video frame based on the plurality of pixel coordinates; and
generating output video data that has a second aspect ratio less than 2:1 and includes the first region of interest in the first video frame.

6. The computer-implemented method of claim 5 , further comprising:
determining first pixel coordinates associated with the first object in the first video frame;
determining a first cropped window from the first video frame, the first cropped window comprising a portion of the first video frame including the first pixel coordinates, wherein the first cropped window:
has a second aspect ratio that is less than 2:1, and
has a first size and a first position within the first video frame;

determining second pixel coordinates associated with the first object in a second video frame of the first video frames;
determining a second cropped window from the second video frame, the second cropped window comprising a portion of the second video frame including the second pixel coordinates, wherein the second cropped window:
has the second aspect ratio, and
has a second size and a second position within th

Assignees

• Amazon Tech Inc

Inventors

• Townsend Matthew Alan
• Bouhnik Moshe
• Kraimer Konstantin
• Oks Eduard

Classifications

• G06T7/73

  using feature-based methods · CPC title

• G11B27/034

  on discs (G11B27/036, G11B27/038 take precedence) · CPC title

• H04N5/2628 Primary

  Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation · CPC title

• G11B27/105

  of operating discs · CPC title

• G06T2207/30168

  Image quality inspection · CPC title