Method and system for content retrieval based on rate-coverage optimization

Patent metadata

Field	Value
Publication number	US-9836535-B2
Application number	US-201514835004-A
Country	US
Kind code	B2
Filing date	Aug 25, 2015
Priority date	Aug 25, 2015
Publication date	Dec 5, 2017
Grant date	Dec 5, 2017

Abstract

A content retrieval method including: extracting a plurality of fingerprints including a plurality of video fingerprints and audio fingerprints from contents stored in a content database; determining representative video fingerprints of the video frames and representative audio fingerprints of the audio sequences; determining a data rate indicating a storage limitation and a coverage indicating a number of searching results to be returned; storing selected representative video fingerprints and representative audio fingerprints based on the storage limitation indicated by the data rate in a fingerprint database; receiving a query containing at least one of video data and audio data submitted by a user; extracting at least one query fingerprint representing the query; determining a number of fingerprints most matching the query fingerprint based on the coverage to generate search results indicating matching contents represented by the number of most matching fingerprints; and returning the search results to the user.

First claim

What is claimed is:

1. A content retrieval method, comprising:
extracting a plurality of fingerprints including a plurality of video fingerprints and a plurality of audio fingerprints from contents stored in a content database, the contents having video frames corresponding to the plurality of video fingerprints and associated audio sequences corresponding to the plurality of audio fingerprints;
determining representative video fingerprints of the video frames and representative audio fingerprints of the audio sequences;
determining a data rate indicating a storage limitation and a coverage indicating a number of searching results to be returned;
storing selected representative video fingerprints and representative audio fingerprints based on the storage limitation indicated by the data rate in a fingerprint database;
receiving a query containing at least one of video data and audio data submitted by a user;
extracting at least one query fingerprint representing the query;
determining a number of fingerprints most matching the at least one query fingerprint based on the coverage to generate search results indicating matching contents represented by the number of most matching fingerprints; and
returning the search results to the user.

2. The content retrieval method according to claim 1 , further comprising:
receiving a selection from the user from the search results; and
retrieving content corresponding to the selection from the content database.

3. The content retrieval method according to claim 1 , wherein:
the at least one query fingerprint including a video query fingerprint and an audio query fingerprint both representing the query; and
the most matching fingerprints are matched with the video query fingerprint or the audio query fingerprint.

4. The content retrieval method according to claim 1 , wherein:
the plurality of video fingerprints are fixed-size feature vectors of the video frames; and
the plurality of audio fingerprints are fixed-number of natural key points of a density distribution of the audio sequences.

5. The content retrieval method according to claim 3 , wherein:
the video query fingerprint and the audio query fingerprint are extracted from the query using same predetermined fingerprint extraction algorithms for extracting the plurality of video fingerprints and the plurality of audio fingerprints from the contents.

6. The content retrieval method according to claim 1 , wherein determining the data rate and the coverage further includes:
determining the data rate indicating the storage limitation and the coverage indicating the number of searching results to be returned based on rate-coverage optimization.

7. The content retrieval method according to claim 6 , wherein determining the data rate and the coverage further includes:
the storage limitation is storage space based on a total number of representative video fingerprints and representative audio fingerprints.

8. The content retrieval method according to claim 6 , wherein:
the coverage indicates, within tolerance of users, a number of searching results to be returned that include a correct search result.

9. The content retrieval method according to claim 6 , wherein:
the rate-coverage optimization is an optimization finding a maximum coverage given a total number of representative fingerprints stored and including a correct search result.

10. The content retrieval method according to claim 9 , wherein:
the rate-coverage optimization is represented by finding:

max

N
V

,

N
A

⁢

(

α
⁢

⁢

f
V

⁡

(

N
V

)

+

(

1
-
α

)

⁢

f
A

⁡

(

N
A

)

)

,

⁢

subject
⁢

⁢
to
⁢

:

⁢

⁢

B
V

×

N
V

+

B
A

×

N
A

≤

R
budget

wherein N V and N A denote number of video representative fingerprints and audio representative fingerprints, respectively; f V (N V ) and f A (N A ) are the optimization processes for video and audio, respectively; αε[0,1]; B V and B A are sizes of each video representative fingerprint and audio representative fingerprint, respectively; and R budget , is the data rate.

11. A non-transitory computer-readable medium having computer program for, when being executed by a processor, performing a content retrieval method, the method comprising:
extracting a plurality of fingerprints including a plurality of video fingerprints and a plurality of audio fingerprints from contents stored in a content database, the contents having video frames corresponding to the plurality of video fingerprints and associated audio sequences corresponding to the plurality of audio fingerprints;
determining representative video fingerprints of the video frames and representative audio fingerprints of the audio sequences;
determining a data rate indicating a storage limitation and a

Assignees

• Tcl Res America Inc

Inventors

• Wang Haohong
• Ning Guanghan
• Zhang Zhi
• Ren Xiaobo

Classifications

• G06F17/30784 Primary

  Physics · mapped topic

• G06F16/434 Primary

  using image data, e.g. images, photos, pictures taken by a user · CPC title

• G06F16/433

  using audio data · CPC title

• G06F16/783 Primary

  using metadata automatically derived from the content · CPC title