Theme classification method based on multimodality, device, and storage medium

Patent metadata

Field	Value
Publication number	US-11995117-B2
Application number	US-202017069410-A
Country	US
Kind code	B2
Filing date	Oct 13, 2020
Priority date	Feb 14, 2020
Publication date	May 28, 2024
Grant date	May 28, 2024

Abstract

A theme classification method based on multimodality is related to a field of a knowledge map. The method includes obtaining text information and non-text information of an object to be classified. The non-text information includes at least one of visual information and audio information. The method also includes determining an entity set of the text information based on a pre-established knowledge base, and then extracting a text feature of the object based on the text information and the entity set. The method also includes determining a theme classification of the object based on the text feature and a non-text feature of the object.

First claim

What is claimed is:

1. A computer-implemented theme classification method based on multimodality, comprising:
extracting, digitally, text information of an object to be classified by using a plurality of computer implemented text-extraction methods, and extracting, digitally, non-text information of the object, wherein the non-text information of the object comprises at least one of visual information and audio information of the object;
determining, digitally, an entity set of the text information by performing entity linking on the text information based on a pre-established knowledge base, wherein performing entity linking comprises linking one or more entities in the entity set involved in the text based on the pre-established knowledge base;
extracting a text feature of the object based on the text information and the entity set;
extracting a non-text feature of the object based on the non-text information; and
determining a theme classification of the object based on the text feature and the non-text feature of the object,
wherein the entity set comprises a plurality of entities involved in the text information, and the method further comprises: for each entity in the entity set,
obtaining a positive example vector based on a text description for the entity and an object related to the entity in the knowledge base;
obtaining a negative example vector based on a randomly sampled text and a randomly sampled object; and
training a vector generation model of the entity based on positive example vectors and negative example vectors, wherein a ratio of a number of the positive example vectors to a number of the negative example vectors is equal to a predetermined ratio.

2. The method of claim 1 , wherein digitally extracting the text information of the object comprises:
extracting an image text by performing optical character recognition for the visual information;
extracting a speech text by performing automatic speech recognition for the audio information;
extracting a character text based on a title and a bullet screen of the object; and
merging the image text, the speech text, and the character text into the text information.

3. The method of claim 1 , wherein determining the theme classification of the object comprises:
based on the visual information and the audio information, extracting a visual feature and an audio feature by an attention-based long-short term memory network;
obtaining a merged feature of the object by merging the visual feature, the audio feature, and the text feature; and
determining multi-level theme classifications of the object based on the merged feature.

4. The method of claim 3 , wherein determining the multi-level theme classifications of the object comprises:
determining a core entity in the entity set; and
determining a theme type corresponding to the core entity based on the knowledge base.

5. The method of claim 4 , wherein determining the multi-level theme classifications of the object further comprises:
obtaining a first feature by weighting the merged feature using a vector of the core entity;
obtaining a second feature by weighting the merged feature using a vector of the theme type; and
obtaining the multi-level theme classifications of the object based on the first feature and the second feature.

6. The method of claim 1 , wherein obtaining the negative example vector comprises:
counting word frequencies of words in text descriptions of all entities in the entity set in the pre-established knowledge base; and
randomly selecting a word sequence used as the negative example vector based on the word frequency of each word.

7. The method of claim 1 , wherein the method further comprises:
determining a theme inference of the object based on the text information; and
determining a merged theme of the object based on the theme classification and the theme inference of the object.

8. An electronic device, comprising:
one or more processors; and
a storage device, configured to store one or more programs,
wherein when the one or more programs are executed by the one or more processors, the electronic device is caused to implement a theme classification method based on multimodality, the method comprising:
extracting, digitally, text information of an object to be classified by using a plurality of computer implemented text-extraction methods, and extracting, digitally, non-text information of the object, wherein the non-text information comprises at least one of visual information and audio information;
determining, digitally, an entity set of the text information by performing entity linking on the text information based on a pre-established knowledge base, wherein performing entity linking comprises linking one or more entities involved in the text based on the pre-established knowledge base;
extracting a text feature of the object based on the text information and the entity set;
extracting a non-text feature of the object based on the non-text information; and
determining a theme classification of the object based on the text feature and the non-text feature of the object,
wherein the entity set comprises a plurality of entities involved in the text information, and the method further comprises: for each entity in the entity set,
obtaining a positive example vector based on a text description for the entity and an object related to the entity in the knowledge base;
obtaining a negative example vector based on a randomly sampled text and a randomly sampled object; and
training a vector generation model of the entity based on positive example vectors and negative example vectors, wherein a ratio of a number of the positive example vectors to a number of the negative example vectors is equal to a predetermined ratio.

9. The electronic device of claim 8 , wherein obtaining the text information of the object comprises:
obtaining an image text by performing optical character recognition for the visual information;
obtaining a speech text by performing automatic speech recognition for the audio information;
obtaining a character text based on a title and a bullet screen of the object; and
obtaining the text information by merging the image text, the speech text, and the character text.

10. The electronic device of claim 8 , wherein determining the theme classification of the object comprises:
based on the visual information and the audio information, extracting a visual feature and an audio feature by an attention-based long-short term memory network;
obtaining a merged feature of the object by merging the visual feature, the audio feature, and the text feature; and
determining multi-level theme classifications of the object based on the merged feature.

11. The electronic device of claim 10 , wherein determining the multi-level theme classifications of the object comprises:
determining a core entity in the entity set; and
determining a theme type corresponding to the core entity based on the knowledge base.

12. The electronic device of claim 11 , wherein determining the multi-level theme classifications of the object further comprises:
obtaining a first feature by weighting the merged feature using a vector of the core entity;
obtaining a second feature by weighting the merged feature using a vector of the theme type; and
obtaining the multi-level theme classifications of the object based on the first feature and the second feature.

13. The electronic device of claim 8 , wherein obtaining the negative example vector comprises:
counting word frequencies of words in text descriptions of all entities of the entity set in the pre-established knowledge b

Assignees

• Beijing Baidu Netcom Sci & Tech Co Ltd

Inventors

• Wang Qi
• Feng Zhifan
• Liu Zhijie
• Chai Chunguang
• Zhu Yong

Classifications

• G06F16/45 Primary

  Clustering; Classification · CPC title

• G06F16/906 Primary

  Clustering; Classification · CPC title

• G06F16/75

  Clustering; Classification · CPC title

• G06F40/279

  Recognition of textual entities · CPC title

• G06F16/36

  Creation of semantic tools, e.g. ontology or thesauri · CPC title