Combining independent solutions to an image or video processing task

Patent metadata

Field	Value
Publication number	US-10824911-B2
Application number	US-201815973184-A
Country	US
Kind code	B2
Filing date	May 7, 2018
Priority date	Jun 8, 2016
Publication date	Nov 3, 2020
Grant date	Nov 3, 2020

Abstract

An algorithm for performing an image or video processing task is generated that may be used to combine a plurality of different independent solutions to the image or video processing task in an optimized manner. A plurality of base algorithms may be applied to a training set of images or video and a first generation of different combining algorithms may be applied to combine the respective solutions from each of the respective base algorithms into respective combined solutions. The respective combined solutions may be evaluated to generate respective fitness scores representing measures of how well the plurality of different combining algorithms each perform the image or video processing task. The algorithms may be iteratively updated to generate an optimized combining algorithm that may be applied to an input image or video.

First claim

The invention claimed is:

1. A method for combining solutions for an image or video processing task, the image or video processing task comprising an automatic highlight detection task, the method comprising:
independently applying a first algorithm and a second algorithm to at least a first portion of a set of input images or video, each of the first and second algorithms comprising an automatic highlight detection algorithm that is different than that of the other;
independently generating respective first and second algorithm results associated with the image or video processing task, the respective first and second algorithm results each comprising data indicative of (i) a temporal reference relating to the occurrence of a highlight within the at least first portion of the set of input images or video, and (ii) a duration of at least one aspect of the highlight;
combining the first algorithm result with the second algorithm result, subsequent to applying one or more of a plurality of modification operations to at least one of the first algorithm result and the second algorithm result, into a first combined solution, the applying of the one or more modification operations comprising applying at least one of: (i) a majority vote for highlights occurring within a given duration, (ii) a logical AND operation for the duration of the at least one aspect of the highlight, or (iii) a logical OR operation for the duration of the at least one aspect of the highlight;
evaluating a plurality of combined solutions, the plurality of combined solutions comprising the first combined solution and at least a second combined solution, the second combined solution being based on independently applying a plurality of algorithms to at least a second portion of the set of input images or video, the evaluating comprising determining a best combined solution for the image or video processing task, the best combined solution meeting at least one predefined optimization criterion with respect to the set of input images or video; and
performing at least the automatic highlight detection task for the set of input images or video based at least on the determined best combined solution.

2. The method of claim 1 , wherein the independently applying of the first plurality of algorithms comprises independently applying two or more of: (1) a motion-based algorithm, (2) a scene change detection algorithm, or (3) a face detection algorithm.

3. The method of claim 2 , wherein the applying of the one or more of the plurality of modification operations comprises applying at least one of: (1) a median filter (MF) operation, (2) an erosion (ERO) operation, or (3) a dilation (DIL) operation.

4. The method of claim 1 , further comprising:
applying one or more of the plurality of modification operations to the first combined solution thereby generating a modified result;
applying the first algorithm to the set of input images or video thereby generating a third algorithm result;
applying one or more of the plurality of modification operations to the third algorithm result thereby generating a first modified result; and
combining the modified result with the first modified result thereby generating a second combined result.

5. The method of claim 4 , further comprising:
applying the second algorithm to the set of input images or video thereby generating a fourth algorithm result;
applying one or more of the plurality of modification operations to the fourth algorithm result thereby generating a second modified result; and
combining the second combined result with the second modified result thereby generating a third combined result.

6. The method of claim 5 , further comprising applying one or more of the plurality of modification operations to the first combined solution, thereby generating a third modified result.

7. The method of claim 6 , further comprising combining the third modified result with the third combined result, the combining of the third modified result with the third combined result resulting in the best combined solution for the respective combined solution the best combined solution meeting the predefined optimization criterion.

8. A non-transitory computer-readable storage apparatus comprising a storage medium having instructions stored thereon, the instructions for combining independent solutions for an image or video processing task, the image or video processing task comprising an automatic highlight detection task, the instructions being configured to, when executed by a processor apparatus, cause a computerized apparatus to:
apply a plurality of automatic highlight detection algorithms to a set of input images or video, each of the plurality of automatic highlight detection algorithms independently generating respective algorithm results to the automatic highlight detection task, each of the respective algorithm results comprising parameters indicating (i) at least one reference to a location in the set of input images or video at which a highlight occurred, and (ii) a duration for at least a portion of the highlight, the respective algorithm results comprising a first algorithm result and a second algorithm result;
combine at least two of the respective algorithm results, subsequent to application of one or more of a plurality of modification operations to at least one of the first algorithm result and the second algorithm result, into a first combined solution;
evaluate at least two combined solutions, the at least two combined solutions comprising the first combined solution, the evaluation to determine a best combined solution, the best combined solution meeting a predefined optimization criterion; and
use the best combined solution to perform at least the automatic highlight detection task for the set of input images or video;
wherein the application of the one or more of the plurality of modification operations comprises application of at least one of (i) a majority vote for hiqhliqhts occurrinq within a prescribed duration, (ii) an AND operation, or (iii) an OR operation.

9. The non-transitory computer-readable storage apparatus of claim 8 , wherein the application of the plurality of automatic highlight detection algorithms comprises an independent application of two or more of: (1) a motion algorithm, (2) a scene change detection algorithm, or (3) a face detection algorithm.

10. The non-transitory computer-readable storage apparatus of claim 9 , wherein:
the application of the one or more modification operations comprises application of at least one of: (1) a median filter (MF) operation, (2) an erosion (ERO) operation, or (3) a dilation (DIL) operation.

11. The non-transitory computer-readable storage apparatus of claim 8 , wherein the instructions are further configured to, when executed by the processor apparatus, cause the computerized apparatus to:
independently apply the plurality of algorithms to the set of input images or video, each of the plurality of algorithms independently generating first respective algorithm results to the automatic highlight detection task; and
combine the first respective algorithm results into a combined solution to the automatic highlight detection task.

12. The non-transitory computer-readable storage apparatus of claim 11 , wherein the at least two algorithm results comprise first algorithm results and second algorithm results, and wherein the instructions are further configured to, when executed by the processor apparatus, cause the computerized apparatus to:
apply one or more of the one or more modification operations to the combined solution thereby resulting in generation of a modified result;
apply a first algorithm of the pl

Assignees

• Gopro Inc

Inventors

• Adsumilli Balineedu Chowdary
• Lustig Ryan

Classifications

• G06N3/126 Primary

  Evolutionary algorithms, e.g. genetic algorithms or genetic programming · CPC title

• G06V10/87

  using selection of the recognition techniques, e.g. of a classifier in a multiple classifier system · CPC title

• G06V10/776

  Validation; Performance evaluation · CPC title

• G06V10/771

  Feature selection, e.g. selecting representative features from a multi-dimensional feature space · CPC title

• G06F18/217

  Validation; Performance evaluation; Active pattern learning techniques · CPC title