System and method for multimedia ranking and multi-modal image retrieval using probabilistic semantic models and expectation-maximization (EM) learning

Patent metadata

Field	Value
Publication number	US-9280562-B1
Application number	US-201213486099-A
Country	US
Kind code	B1
Filing date	Jun 1, 2012
Priority date	Jan 31, 2006
Publication date	Mar 8, 2016
Grant date	Mar 8, 2016

Abstract

Systems and Methods for multi-modal or multimedia image retrieval are provided. Automatic image annotation is achieved based on a probabilistic semantic model in which visual features and textual words are connected via a hidden layer comprising the semantic concepts to be discovered, to explicitly exploit the synergy between the two modalities. The association of visual features and textual words is determined in a Bayesian framework to provide confidence of the association. A hidden concept layer which connects the visual feature(s) and the words is discovered by fitting a generative model to the training image and annotation words. An Expectation-Maximization (EM) based iterative learning procedure determines the conditional probabilities of the visual features and the textual words given a hidden concept class. Based on the discovered hidden concept layer and the corresponding conditional probabilities, the image annotation and the text-to-image retrieval are performed using the Bayesian framework.

First claim

The invention claimed is:

1. A method of ranking multimedia works with respect to a query, comprising:
(a) storing, in a memory, information defining a conditional probability framework representing a plurality of concepts represented within at least two different media types of multimedia works comprising semantic concepts and visual feature concepts, based on analysis of at least a subset of the plurality of works;
(b) receiving a query comprising information defining at least one of the semantic concepts and the visual feature concepts;
(c) automatically determining a set of multimedia works of the at least two different media types corresponding to the at least one of said concepts associated with the query by associating semantic concepts of a first multimedia work with visual feature concepts of a second multimedia work as annotations, using a probabilistic framework comprising a visual feature layer, a semantic layer, and a hidden concept layer which connects the visual feature layer and the semantic layer, with at least one automated processor, wherein conditional probabilities of the visual feature concepts and the annotations given a hidden concept class are determined based on an Expectation-Maximization (EM) based iterative learning procedure;
(d) automatically ranking the determined set of multimedia works of the at least two different media types determined to correspond to the at least one of said concepts associated with the query in accordance with a relevance to the query according to a joint probability distribution which models a probability that the query associated with the at least one of said concepts is associated with a respective multimedia work, with the at least one automated processor; and
(e) at least one of (i) storing in the memory, and (ii) outputting information selectively dependent on the ranking.

2. The method according to claim 1 , the at least one of said concepts associated with the query is implicit; further comprising automatically mapping multimedia works to the conditional probability framework, using at least one processor, based on implicit concepts represented within each multimedia work, the probabilistic framework comprising at least one joint probability distribution which models a probability that a symbol belonging to a respective concept is an annotation symbol of a respective multimedia work.

3. The method according to claim 1 , wherein the conditional probability framework is organized according to semantic concepts, and the query comprises a multimedia work having implicit semantic concepts expressed in a non-semantic form.

4. The method according to claim 1 , wherein the query comprises an image, and wherein the image is processed to determine implicit semantic image content characteristics.

5. The method according to claim 1 , wherein at least one semantic word is associated with the received query as an implicit semantic concept, the probabilistic framework is a semantic probabilistic framework in which multimedia works are mapped to semantic concepts, and the at least one semantic word is used to search the mapped multimedia works within the probabilistic semantic framework.

6. The method according to claim 5 , wherein a plurality of words belong to at least one respective semantic concept, such that the association of the respective multimedia work with the plurality of semantic concepts is through automatically generated annotation words and analysis of the joint probability distribution.

7. The method according to claim 1 , wherein the query comprises an image, said at least two different media types comprise multimedia works comprising words and multimedia works comprising images having visual features, and said automatically determining comprises associating words with a respective image using a Bayesian model comprising the hidden concept layer which connects the visual feature layer and the word layer, which is discovered by fitting the generative model to a training set comprising images and annotation words, wherein the conditional probabilities of the visual features and the annotation words given the hidden concept class are determined based on the Expectation-Maximization (EM) based iterative learning procedure.

8. The method according to claim 7 , wherein f i , i ε[1,N] denotes a visual feature vector of images in a training database, where N is the size of the database, w j , jε[1,M] denotes the distinct textual words in a training annotation word set, where M is the size of annotation vocabulary in the training database, the visual features of images in the database, f i =[f i 1 ,f i 2 , . . . f i L ], iε[1, N] are known i.i.d. samples from an unknown distribution, having a visual feature dimension L, the specific visual feature annotation word pairs (f i , w j ),iε[1, N], jε[1,M] are known i.i.d. samples from an unknown distribution, associated with an unobserved semantic concept variable zε Z={z 1 , . . . z k }, in which each observation of one visual feature fε F={f i ,f 2 , . . . , f N } belongs to one or more concept classes z k and each observation of one word wε V={w 1 , w 2 , . . . ,w M } in one image f i belongs to one concept class, in which the observation pairs (f i ,w j ) are assumed to be generated independently, and the pairs of random variables (f i ,w j ) are assumed to be conditionally independent given the respective hidden concept z k , such that P(f i ,w j |z k )= (f i |z k )P V (w j |z k );
the visual feature and word distribution is treated as a randomized data generation process, wherein a probability of a concept is represented as P z (z k );
a visual feature is selected f i εF with probability (f i |z k ); and a textual word is selected w j εV with probability P V (w j |z k ), from which an observed pair (f i ,w j ) is obtained, such that a joint probability model is expressed as follows:

P
⁡

(

f
i

,

w
j

)

=

⁢

P
⁡

(

w
j

)

⁢
⁢

(

f
i

❘

w
j

)

=

⁢

P
⁡

(

Assignees

• Zhang Ruofei
• Zhang Zhongfei
• Univ New York State Res Found

Inventors

• Zhang Ruofei
• Zhang Zhongfei

Classifications

• G06N5/02

  Knowledge representation; Symbolic representation · CPC title

• G06F16/5838 Primary

  using colour · CPC title

• G06N7/01

  Probabilistic graphical models, e.g. probabilistic networks · CPC title

• G06F18/24155

  Bayesian classification · CPC title

• G06F16/5846 Primary

  using extracted text · CPC title