Interacting with hierarchical clusters of video segments using a metadata search

Patent metadata

Field	Value
Publication number	US-11880408-B2
Application number	US-202017017370-A
Country	US
Kind code	B2
Filing date	Sep 10, 2020
Priority date	Sep 10, 2020
Publication date	Jan 23, 2024
Grant date	Jan 23, 2024

Abstract

Embodiments are directed to techniques for interacting with a hierarchical video segmentation by performing a metadata search. Generally, various types of metadata can be extracted from a video, such as a transcript of audio, keywords from the transcript, content or action tags visually extracted from video frames, and log event tags extracted from an associated temporal log. The extracted metadata is segmented into metadata segments and associated with corresponding video segments defined by a hierarchical video segmentation. As such, a metadata search can be performed to identify matching metadata segments and corresponding matching video segments defined by a particular level of the hierarchical segmentation. Matching metadata segments are emphasized in a composite list of the extracted metadata, and matching video segments are emphasized on the video timeline. Navigating to a different level of the hierarchy transforms the search results into corresponding coarser or finer segments defined by the level.

First claim

What is claimed is:

1. One or more computer storage media storing computer-useable instructions that, when used by one or more computing devices, cause the one or more computing devices to perform operations comprising:
accessing a hierarchical segmentation of a video timeline of a video, the hierarchical segmentation associating extracted metadata extracted from the video by one or more machine learning models with corresponding video segments defined by a first level of the hierarchical segmentation;
receiving, via a search bar of a user interface, an input identifying a textual search criterion;
executing a search of the extracted metadata using the textual search criterion to identify matching metadata segments of the extracted metadata and corresponding matching video segments of video segments defined by the first level of the hierarchical segmentation; and
causing the user interface to visually emphasize, on the video timeline, the corresponding matching video segments that are from the first level and correspond to the matching metadata segments that match the input in the search bar.

2. The one or more computer storage media of claim 1 , wherein causing the user interface to visually emphasize the matching video segments comprises animating the corresponding matching video segments on the video timeline by inducing a transient oscillating displacement of representations of the corresponding matching video segments on the video timeline.

3. The one or more computer storage media of claim 1 , wherein causing the user interface to visually emphasize the matching video segments comprises animating the corresponding matching video segments on the video timeline by inducing a traveling wave that displaces representations of the corresponding matching video segments on the video timeline as the traveling wave travels down the video timeline.

4. The one or more computer storage media of claim 1 , the operations further comprising causing the user interface to visually emphasize the matching metadata segments on a composite list of the extracted metadata segmented at locations in the composite list corresponding to boundaries of the corresponding video segments defined by the first level of the hierarchical segmentation.

5. The one or more computer storage media of claim 1 , the operations further comprising causing the user interface to visually emphasize the matching metadata segments in a metadata panel and to visually emphasize the corresponding matching video segments on the video timeline using a same type of visual emphasis.

6. The one or more computer storage media of claim 1 , the operations further comprising:
causing the user interface to segment, in response to an input navigating from the first level to a different level of the hierarchical segmentation, a composite list of the extracted metadata into an updated set of metadata segments, the composite list segmented at locations in the composite list corresponding to boundaries of a second set of video segments defined by the different level of the hierarchical segmentation;
identifying an updated set of matching metadata segments, from the updated set of metadata segments defined by the different level, that match the search criterion; and
causing the user interface to visually emphasize on the video timeline an updated set of matching video segments, of the second set of video segments defined by the different level, corresponding to the updated set of matching metadata segments.

7. The one or more computer storage media of claim 1 , wherein the extracted metadata comprises transcribed audio of the corresponding video segments.

8. The one or more computer storage media of claim 1 , wherein the extracted metadata comprises tags visually extracted from video frames of the corresponding video segments.

9. The one or more computer storage media of claim 1 , wherein the extracted metadata comprises log event tags extracted from a temporal log associated with the video.

10. The one or more computer storage media of claim 1 , the operations further comprising detecting the input selecting a metadata tag as the search criterion from a popup list of top metadata tags in the extracted metadata.

11. A computerized method comprising:
accessing a hierarchical segmentation of a video timeline of a video, the hierarchical segmentation associating extracted metadata extracted from the video by one or more machine learning models with corresponding video segments defined by a level of the hierarchical segmentation;
receiving, via a search bar of a user interface, an input identifying a keyword;
executing a search of the extracted metadata for the keyword to identify matching metadata segments of the extracted metadata and corresponding matching video segments of video segments defined by the level of the hierarchical segmentation; and
causing the user interface to update, on the video timeline, a visual representation of the corresponding matching video segments that are from the level and correspond to the matching metadata segments that match the input in the search bar.

12. The computerized method of claim 11 , wherein causing the user interface to update the visual representation of the corresponding matching video segments comprises animating the corresponding matching video segments on the video timeline by inducing a transient oscillating displacement of representations of the corresponding matching video segments on the video timeline.

13. The computerized method of claim 11 , causing the user interface to update the visual representation of the corresponding matching video segments comprises animating the corresponding matching video segments on the video timeline by inducing a traveling wave that displaces representations of the corresponding matching video segments on the video timeline as the traveling wave travels down the video timeline.

14. The computerized method of claim 11 , further comprising causing the user interface to visually emphasize the matching metadata segments on a composite list of the extracted metadata segmented at locations in the composite list corresponding to boundaries of the corresponding video segments defined by the level of the hierarchical segmentation.

15. The computerized method of claim 11 , further comprising causing, in response to an input navigating from the level to a different level of the hierarchical segmentation, the user interface to visually emphasize an updated set of matching video segments defined by the different level and corresponding to an updated set of matching metadata segments that are defined by the different level and match the input in the search bar.

16. The computerized method of claim 11 , further comprising detecting the input identifying the keyword from a popup list of top metadata tags in the extracted metadata.

17. A computer system comprising:
one or more hardware processors and memory configured to provide computer program instructions to the one or more hardware processors;
a video interaction engine configured to use the one or more hardware processors to perform operations comprising:
receiving a keyword via a search bar of a user interface;
executing a search for the keyword in extracted metadata extracted from a video by one or more machine learning models to identify matching metadata segments of the extracted metadata and corresponding matching video segments of video segments defined by a level of a hierarchical segmentation of a video timeline of the video; and
causing at least one of (i) the user interface to visually emphasize on the vi

Assignees

• Adobe Inc

Inventors

• Walker Seth
• Kim Joy Oakyung
• Evans Morgan Nicole
• Yoo Najika Skyler Halsema
• Agarwala Aseem
• Brandt Joel R
• Popović Jovan
• Dontcheva Lubomira
• Li Dingzeyu
• Shin Hijung
• Bai Xue

Classifications

• G06F16/738 Primary

  Presentation of query results · CPC title

• G06F3/0482

  Interaction with lists of selectable items, e.g. menus · CPC title

• G06F3/0484

  for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range · CPC title

• G06F16/735

  Filtering based on additional data, e.g. user or group profiles · CPC title

• G06F16/743

  a collection of video files or sequences · CPC title