Providing context aware video searching

Patent metadata

Field	Value
Publication number	US-10998007-B2
Application number	US-201916588662-A
Country	US
Kind code	B2
Filing date	Sep 30, 2019
Priority date	Sep 30, 2019
Publication date	May 4, 2021
Grant date	May 4, 2021

Abstract

This disclosure relates to methods, non-transitory computer readable media, and systems that can generate a context-aware-video-progress bar including a video-scene-proportionate timeline with time-interval sections sized according to relative scene proportions within time intervals of a video. In some implementations, for instance, the disclosed systems determine relative proportions of scenes within a video across time intervals of the video and generate a video-scene-proportionate timeline comprising time-interval sections sized proportionate to the relative proportions of scenes across the time intervals. By integrating the video-scene-proportionate timeline within a video-progress bar, the disclosed systems generate a context-aware-video-progress bar for a video. Such a context-aware-video-progress bar can facilitate more precise and intelligent scrubbing through a video, a dynamic graphical user interface for navigating within and identifying frames of the video, and a flexible user-friendly tool for quickly identifying scenes.

First claim

We claim:

1. A non-transitory computer readable medium storing instructions thereon that, when executed by at least one processor, cause a computing device to:
detect reference frames comprising image features similar to other frames of scenes within a video by:
determining, utilizing a feature-detection algorithm, a numerical value of image-feature points that are shared in common between particular frames of same scenes and are associated with objects from the particular frames;
generating feature-matching scores for frames corresponding to the scenes whereby a feature-matching score indicates a degree to which particular image-feature points within a particular frame matches image-feature points within the other frames of a same scene based on the numerical value of image-feature points that are shared in common between the particular frame and the other frames of the same scene; and
selecting particular reference frames for particular scenes based on a highest feature-matching score from among the feature-matching scores for frames corresponding to a particular scene;

determine relative proportions of the reference frames across a set of time intervals within the video;
generate a video-scene-proportionate timeline for the video comprising time-interval sections for the set of time intervals sized according to the relative proportions of the reference frames across the set of time intervals; and
provide, for display within a graphical user interface, the video and a context-aware-video-progress bar comprising the video-scene-proportionate timeline.

2. The non-transitory computer readable medium of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to detect the reference frames comprising the image features similar to the other frames of the scenes within the video in part by:
detecting, for a first scene, a first reference frame comprising a subset of image-feature points depicting a first object; and
detecting, for a second scene, a second reference frame comprising a subset of image-feature points depicting a second object.

3. The non-transitory computer readable medium of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to identify a subset of frames corresponding to a scene within the video in part by:
detecting a set of image-feature points within a set of frames corresponding to the video;
comparing a selected subset of image-feature points from a selected frame to a subset of image-feature points from each frame of a contiguous set of frames within a threshold number of frames from the selected frame;
determining that the selected subset of image-feature points from the selected frame does not satisfy a minimum number or a minimum percentage of matching image-feature points shared in common with the subset of image-feature points from each frame of the contiguous set of frames; and
identifying the selected frame as a frame for the subset of frames corresponding to the scene.

4. The non-transitory computer readable medium of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to generate a feature-matching score for a frame from a subset of feature-matching scores for a subset of frames by determining a mean number of matching image-feature points for the frame in common with each other frame within the subset of frames.

5. The non-transitory computer readable medium of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to determine a normalized congestion score for a time interval from the set of time intervals by determining a ratio of a number of the reference frames within the time interval to a total number of the reference frames within the video.

6. The non-transitory computer readable medium of claim 5 , further comprising instructions that, when executed by the at least one processor, cause the computing device to generate the video-scene-proportionate timeline for the video by:
generating a first time-interval section sized in proportion to a first normalized congestion score for a first time interval of the video; and
generating a second time-interval section sized in proportion to a second normalized congestion score for a second time interval of the video.

7. The non-transitory computer readable medium of claim 6 , further comprising instructions that, when executed by the at least one processor, cause the computing device to determine the numerical value of image-feature points that are shared in common between the particular frames of the same scenes in part by:
resizing a subset of frames corresponding to a scene to generate a subset of smaller frames;
detecting, utilizing the feature-detection algorithm, subsets of image-feature points within particular smaller frames from the subset of smaller frames; and
determining a particular numerical value of image-feature points shared in common between the particular smaller frames.

8. The non-transitory computer readable medium of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to:
detect a user selection of a time location within the context-aware-video-progress bar corresponding to a time-interval section sized in proportion to a congestion score for a time interval within the video-scene-proportionate timeline; and
based on detecting the user selection of the time location:
select a portion of the video from a time within the time interval corresponding to the time-interval section of the video-scene-proportionate timeline; and
render a thumbnail image of the portion of the video overlaid on a display of the video.

9. A system comprising:
one or more memory devices comprising a video having multiple frames; and
one or more server devices that cause the system to:
detect reference frames comprising image features similar to other frames of scenes within a video by:
determining, utilizing a feature-detection algorithm, a numerical value of image-feature points that are shared in common between particular frames of same scenes and are associated with objects from the particular frames;
generating feature-matching scores for frames corresponding to the scenes whereby a feature-matching score indicates a degree to which particular image-feature points within a particular frame matches image-feature points within the other frames of a same scene based on the numerical value of image-feature points that are shared in common between the particular frame and the other frames of the same scene; and
selecting particular reference frames for particular scenes from among the frames based on a highest feature-matching score from among the feature-matching scores for frames corresponding to a particular scene;

determine relative proportions of the reference frames across a set of time intervals within the video by:
determining a number of the reference frames within each time interval from the set of time intervals;
generating a set of congestion scores for the set of time intervals indicating relative proportions of the reference frames within particular time intervals;

generate a video-scene-proportionate timeline for the video comprising time-interval sections in proportion to the set of congestion scores for the set of time intervals; and
provide, for display within a graphical user interface, the video and a context-aware-video-progress bar comprising the video-scene-proportionate timeline.

10. The system of claim 9 , wherein the one or more server devices further cau

Assignees

• Adobe Inc

Inventors

• Bedi Ajay
• Jindal Amol

Classifications

• G06V10/462

  Salient features, e.g. scale invariant feature transforms [SIFT] · CPC title

• G06V10/761

  Proximity, similarity or dissimilarity measures · CPC title

• G06F18/22

  Matching criteria, e.g. proximity measures · CPC title

• G11B27/34 Primary

  Indicating arrangements  {(indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D)} · CPC title

• G06V20/46

  Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames · CPC title