Method and a system for multimodal search key based multimedia content extraction

Patent metadata

Field	Value
Publication number	US-2020311123-A1
Application number	US-201916368136-A
Country	US
Kind code	A1
Filing date	Mar 28, 2019
Priority date	Mar 28, 2019
Publication date	Oct 1, 2020
Grant date	—

Abstract

A method and a system are described for multimodal search key based multimedia content extraction. The method includes receiving a multimedia content search request for a multimedia content, where the search request includes multimodal inputs in one or more fragments. The one or more fragments are interleaved to a composite fragment by removing overlapping content from the one or more fragments. The composite fragment is tagged to one or more attributes associated with the multimedia content based on a deep-learning of a context associated with the composite fragment. The method further includes identifying from a multimedia content database a correlation between the composite fragment and the multimedia content based on the context, where the multimedia content is stored in the multimedia content database. The method includes extracting the multimedia content identified from the multimedia content database.

First claim

We claim:

1 . A method of multi-modal search key based multimedia content extraction, the method comprising:
receiving, by a search key generator device, a multimedia content search request for a multimedia content, wherein the multimedia content search request comprises multimodal inputs in one or more fragments and wherein the one or more fragments comprises at least one of a text fragment, an image fragment, an audio fragment and a video fragment; interleaving, by the search key generator device, the one or more fragments to a composite fragment by removing overlapping content from the one or more fragments; tagging, by the search key generator device, the composite fragment to one or more attributes associated with the multimedia content based on a deep-learning of a context associated with the composite fragment; identifying, by the search key generator device, from a multimedia content database a correlation between the composite fragment and the multimedia content based on the context, wherein the multimedia content is stored in the multimedia content database, and wherein searching for the correlation comprises comparing the context of the composite fragment to a context of the multimedia content; and extracting, by the search key generator device, the multimedia content identified from the multimedia content database based on comparing the context of the composite fragment to the context of the multimedia content.

2 . The method of claim 1 , wherein one of the multimodal inputs further acts as a modifier to the multimedia content search request.

3 . The method of claim 1 , wherein the one or more attributes associated with the multimedia content comprises at least one of a name, a name of a place, a color, a verb, and an adjective.

4 . The method of claim 1 , wherein the deep learning further comprises:
determining the one or more attributes associated with the multimedia content; populating the multimedia content database based on the one or more attributes; assigning a common label to the composite fragment based on the common context; denominating weightages to the one or more fragments in the composite fragment based on the multimedia content search request, the attributes and the context; and altering the weightages based on receiving the one of the multimodal inputs multimodal inputs as modifier.

5 . The method of claim 1 , wherein extraction comprises formatting the multimedia content extracted, wherein formatting further comprises at least synchronizing the audio fragment with the video fragment, and associating the text format and the image format to the synchronized audio format and video format.

6 . A search key generator device for recommending products to a user comprising:
a processor; and a memory communicatively coupled to the processor, wherein the memory stores processor executable instructions, which on execution causes the processor to:
receive a multimedia content search request for a multimedia content, wherein the multimedia content search request comprises multimodal inputs in one or more fragments and wherein the one or more fragments comprises at least one of a text fragment, an image fragment, an audio fragment and a video fragment;
interleave the one or more fragments to a composite fragment by removing overlapping content from the one or more fragments;
tag the composite fragment to one or more attributes associated with the multimedia content based on a deep-learning of a context associated with the composite fragment;
identify from a multimedia content database a correlation between the composite fragment and the multimedia content based on the context, wherein the multimedia content is stored in the multimedia content database, and wherein searching for the correlation comprises comparing the context of the composite fragment to a context of the multimedia content; and
extract the multimedia content identified from the multimedia content database based on comparing the context of the composite fragment to the context of the multimedia content.

7 . The search key generator device of claim 6 , wherein one of the multimodal inputs further acts as a modifier to the multimedia content search request.

8 . The search key generator device of claim 6 , wherein the one or more attributes associated with the multimedia content comprises at least one of a name, a name of a place, a color, a verb, and an adjective.

9 . The search key generator device of claim 6 , wherein the deep learning further comprises:
determining the one or more attributes associated with the multimedia content; populating the multimedia content database based on the one or more attributes; assigning a common label to the composite fragment based on the common context; denominating weightages to the one or more fragments in the composite fragment based on the multimedia content search request, the attributes and the context; and altering the weightages based on receiving the one of the multimodal inputs multimodal inputs as modifier.

10 . The search key generator device of claim 6 , wherein extraction comprises formatting the multimedia content extracted, wherein formatting further comprises at least synchronizing the audio fragment with the video fragment, and associating the text format and the image format to the synchronized audio format and video format.

11 . A non-transitory computer-readable storage medium having stored thereon, a set of computer-executable instructions for causing a computer comprising one or more processors to perform steps comprising:
receiving, by a search key generator device, a multimedia content search request for a multimedia content, wherein the multimedia content search request comprises multimodal inputs in one or more fragments and wherein the one or more fragments comprises at least one of a text fragment, an image fragment, an audio fragment and a video fragment; interleaving, by the search key generator device, the one or more fragments to a composite fragment by removing overlapping content from the one or more fragments; tagging, by the search key generator device, the composite fragment to one or more attributes associated with the multimedia content based on a deep-learning of a context associated with the composite fragment; identifying, by the search key generator device, from a multimedia content database a correlation between the composite fragment and the multimedia content based on the context, wherein the multimedia content is stored in the multimedia content database, and wherein searching for the correlation comprises comparing the context of the composite fragment to a context of the multimedia content; and extracting, by the search key generator device, the multimedia content identified from the multimedia content database based on comparing the context of the composite fragment to the context of the multimedia content.

Assignees

• Wipro Ltd

Inventors

• Ramachandra Iyer Manjunath

Classifications

• G06N3/08

  Learning methods · CPC title

• G06F16/43 Primary

  Querying · CPC title

• G06Q30/0631

  Recommending goods or services · CPC title

• G06F16/432

  Query formulation · CPC title

• G06F16/41

  Indexing; Data structures therefor; Storage structures · CPC title