Focalized summarizations of a video stream

Patent metadata

Field	Value
Publication number	US-10719552-B2
Application number	US-201815916997-A
Country	US
Kind code	B2
Filing date	Mar 9, 2018
Priority date	Mar 2, 2017
Publication date	Jul 21, 2020
Grant date	Jul 21, 2020

Abstract

The disclosure includes a system and method for creating, storing, and retrieving a focalized visualization related to a location, an event or a subject of interest. A visualization server receives a query for creating a visualization from a client device, identifies and retrieves one or more segments of a video stream satisfying the query, and generates the visualization based on the one or more segments of the video stream.

First claim

What is claimed is:

1. A computer-implemented method comprising:
extracting a plurality of salient fragments of a video, each salient fragment including one or more segments of the video, each segment being associated with segment metadata, the segment metadata including relationship information between segments based on a cause-and-effect relationship;
building a database of the plurality of salient fragments, the database storing records of each segment of the video, each record including a location at which a respective segment is stored and the associated segment metadata;
indexing the segments of the video in the database based on the cause-and-effect relationship information in the segment metadata, wherein the cause-and-effect relationship information relates a first segment to a second segment in response to an object or an action in the first segment having a cause-and-effect relationship with an object or an action in the second segment;
receiving a query for creating a visualization from a client device, wherein the query is generated in response to a trigger event from an external system and the query includes a set of restrictions related to the trigger event; and
responsive to the query, performing a visualization creation job by:
generating a URL to show a state of the visualization creation job;
identifying related salient fragments of the video from the database that are related by the cause-and-effect relationship between the segments based on the set of restrictions;
retrieving one or more identified salient fragments that satisfy the set of restrictions included in the query;
generating the visualization based on the one or more salient fragments of the video; and
providing the generated visualization at the URL as a result of the visualization creation job.

2. The computer-implemented method of claim 1 , wherein the set of restrictions includes a motion threshold.

3. The computer-implemented method of claim 1 , further comprising:
receiving, by an application server, the trigger event from the client device; and
converting, by the application server, the trigger event to the query for the visualization.

4. The computer-implemented method of claim 3 , wherein converting, by the application server, the trigger event to the query for the visualization further comprises specifying the set of restrictions related to the trigger event for the query, and wherein the set of restrictions includes at least one of a temporal parameter, a spatial parameter, and a saliency attribute.

5. The computer-implemented method of claim 1 , further comprising:
dispatching the visualization creation job of creating the visualization to a visualization worker based on a job queue; and
communicating a request, by the visualization worker, to a segment server to identify the related salient fragments of the video from the database and retrieve the one or more identified salient fragments that satisfy the set of restrictions included in the query.

6. The computer-implemented method of claim 1 , wherein each salient fragment includes a subset of frames and a subset of pixels from the subset of frames.

7. The computer-implemented method of claim 6 , further comprising:
grouping, by a middleware server, the one or more segments into topics based on identifiers associated with each segment of the one or more segments;
configuring, by the middleware server, one or more indexers to run as consumer groups subscribed to the topics;
identifying, by the middleware server, from the one or more indexers, an indexer to store a segment;
adding, by the indexer, the record of the segment to the database based on the associated segment metadata; and
archiving, by the indexer, the segment in a persistent storage.

8. The computer-implemented method of claim 1 , wherein retrieving the one or more identified salient fragments that satisfy the set of restrictions included in the query comprises establishing a tunnel to communicate with a segment server to retrieve the one or more identified salient fragments.

9. A system comprising:
one or more processors; and
a memory, the memory storing instructions, which when executed cause the one or more processors to:
extract a plurality of salient fragments of a video, each salient fragment including one or more segments of the video, each segment being associated with segment metadata, the segment metadata including relationship information between segments based on a cause-and-effect relationship;
build a database of the plurality of salient fragments, the database storing records of each segment of the video, each record including a location at which a respective segment is stored and the associated segment metadata;
index the segments of the video in the database based on the cause-and-effect relationship information in the segment metadata, wherein the cause-and-effect relationship information relates a first segment to a second segment in response to an object or an action in the first segment having a cause-and-effect relationship with an object or an action in the second segment;
receive a query creating a visualization from a client device, wherein the query is generated in response to a trigger event from an external system and the query includes a set of restrictions related to the trigger event; and
responsive to the query, perform a visualization creation job by:
generating a URL to show a state of the visualization creation job;
identifying related salient fragments of the video from the database that are related by the cause-and-effect relationship between the segments based on the set of restrictions;
retrieving one or more identified salient fragments that satisfy the set of restrictions included in the query;
generating the visualization based on the one or more salient fragments of the video; and
providing the generated visualization at the URL as a result of the visualization creation job.

10. The system of claim 9 , wherein the set of restrictions includes a motion threshold.

11. The system of claim 9 , wherein the instructions further cause the one or more processors to:
receive the trigger event from the client device; and
convert the trigger event to the query for the visualization.

12. The system of claim 11 , wherein to convert, by the application server, the trigger event to the query for the visualization, the instructions further cause the one or more processors to:
specify the set of restrictions related to the trigger event for the query; and
wherein the set of restrictions includes at least one of a temporal parameter, a spatial parameter, and a saliency attribute.

13. The system of claim 9 , wherein the instructions cause the one or more processors to:
dispatch the visualization creation job of creating the visualization to a visualization worker based on a job queue; and
communicate a request, by the visualization worker, to a segment server to identify the related salient fragments of the video from the database and retrieve the one or more identified salient fragments that satisfy the set of restrictions included in the query.

14. The system of claim 9 , wherein each salient fragment includes a subset of frames and a subset of pixels from the subset of frames.

15. The system of claim 14 , wherein the instructions cause the one or more processors to:
group the one or more segments into topics based on identifiers associated with each segment of the one or more segments;
configure one or more indexers to run as consumer groups subscribed to the topics;
identify from the one or more indexers, an indexer to store a

Assignees

• Moraleda Jorge
• Gonzalez Banos Hector H
• Covarrubias Marco Antonio
• Ricoh Co Ltd

Inventors

• Moraleda Jorge
• Gonzalez-Banos Hector H
• Covarrubias Marco Antonio

Classifications

• G06V10/62

  relating to a temporal dimension, e.g. time-based feature extraction; Pattern tracking · CPC title

• G06V10/40

  Extraction of image or video features · CPC title

• G06V20/47

  Detecting features for summarising video content · CPC title

• G06V40/173

  face re-identification, e.g. recognising unknown faces across different face tracks · CPC title

• G06V20/49

  Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes · CPC title