Accurate tag relevance prediction for image search

Patent metadata

Field	Value
Publication number	US-11816888-B2
Application number	US-202016853111-A
Country	US
Kind code	B2
Filing date	Apr 20, 2020
Priority date	Feb 12, 2016
Publication date	Nov 14, 2023
Grant date	Nov 14, 2023

Abstract

Embodiments of the present invention provide an automated image tagging system that can predict a set of tags, along with relevance scores, that can be used for keyword-based image retrieval, image tag proposal, and image tag auto-completion based on user input. Initially, during training, a clustering technique is utilized to reduce cluster imbalance in the data that is input into a convolutional neural network (CNN) for training feature data. In embodiments, the clustering technique can also be utilized to compute data point similarity that can be utilized for tag propagation (to tag untagged images). During testing, a diversity based voting framework is utilized to overcome user tagging biases. In some embodiments, bigram re-weighting can down-weight a keyword that is likely to be part of a bigram based on a predicted tag set.

First claim

What is claimed is:

1. A computer-implemented method comprising:
inputting an image into a trained image classifier, wherein the trained image classifier is based on a recursive clustering process that generates data clusters, wherein a step in the recursive clustering process comprises partitioning data into clusters and recombining all clusters from the clusters with a size exceeding a predefined threshold prior to a subsequent partitioning;
determining, for the image, a related data cluster from the data clusters of the trained image classifier, the related data cluster comprising a set of training images and corresponding tags; and
propagating a tag to the image, the tag selected from the corresponding tags of the set of training images.

2. The method of claim 1 , wherein the related data cluster is identified using a similarity analysis of features in the image in relation to the data clusters.

3. The method of claim 1 , wherein the tag is selected from the corresponding tags of the set based on a relevance score.

4. The method of claim 3 , wherein the relevance score is determined using a confidence value of the tag based on a measure of similarity between two of the corresponding tags in the related data cluster, wherein the tag is one of the two corresponding tags.

5. The method of claim 4 , wherein determining the measure of similarity between the two corresponding tags in the cluster further comprises finding a cluster-sensitive hashing distance representing a number of times the two corresponding tags are assigned to a same cluster in relation to iterations of the recursive clustering process.

6. The method of claim 3 , wherein the relevance score comprises a confidence value of the tag and down-weighting the tag when the tag is determined to be part of a bigram based on a predicted tag set.

7. The method of claim 3 , wherein the relevance score is determined by assigning an effective number of votes to the tag based on a first similarity between tag sequences for a first image and a second image in relation to a second similarity between a first image tag provider of the tag and a second tag provider of a second tag.

8. The method of claim 1 , wherein the tag selected from the corresponding tags has a highest relevance score.

9. The method of claim 1 , further comprising:
using a support vector machine (SVM) classifier for a dictionary of tags to determine a confidence value for the tag, wherein the SVM classifier is trained in a lower dimensional space using k nearest neighbors as training samples.

10. A non-transitory computer storage medium comprising computer-useable instructions that, when used by one or more computing devices, cause the one or more computing devices to perform operations comprising:
inputting an image into a trained image classifier, wherein the trained image classifier is based on a recursive clustering process that generates data clusters, wherein a step in the recursive clustering process comprises partitioning data into clusters and recombining all clusters from the clusters with a size exceeding a predefined threshold prior to a subsequent partitioning;
determining, for the image, a related data cluster from the data clusters of the trained image classifier, the related data cluster comprising a set of training images and corresponding tags, the related data cluster identified from a similarity analysis of features in the image in relation to the data clusters; and
propagating a tag to the image, the tag selected from the corresponding tags of the set of training images.

11. The non-transitory computer storage medium of claim 10 , wherein the tag is selected from the corresponding tags of the set based on a relevance score.

12. The non-transitory computer storage medium of claim 11 , wherein the relevance score is determined using a confidence value of the tag based on a measure of similarity between two of the corresponding tags in the related data cluster wherein the tag is one of the two corresponding tags.

13. The non-transitory computer storage medium of claim 12 , wherein determining the measure of similarity between the two corresponding tags in the cluster further comprises finding a cluster-sensitive hashing distance representing a number of times the two corresponding tags are assigned to a same cluster in relation to iterations of the recursive clustering process.

14. The non-transitory computer storage medium of claim 11 , wherein the relevance score comprises a confidence value of the tag and the tag is down-weighted when determined to be part of a bigram based on a predicted tag set.

15. The non-transitory computer storage medium of claim 11 , wherein the relevance score is determined by assigning an effective number of votes to the tag based on a first similarity between tag sequences for a first image and a second image in relation to a second similarity between a first image tag provider of the tag and a second tag provider of a second tag.

16. The non-transitory computer storage medium of claim 10 , wherein the tag selected from the corresponding tags has a highest relevance score.

17. The non-transitory computer storage medium of claim 10 , further comprising:
using a support vector machine (SVM) classifier for a dictionary of tags to determine a confidence value for the tag, wherein the SVM classifier is trained in a lower dimensional space using k nearest neighbors as training samples.

18. A computing system comprising:
means for inputting an image into a trained image classifier, wherein the trained image classifier is based on a recursive clustering process that generates data clusters, wherein a step in the recursive clustering process comprises partitioning data into clusters and recombining all clusters from the clusters with a size exceeding a predefined threshold prior to a subsequent partitioning;
means for determining, for the image, a related data cluster from the data clusters of the trained image classifier, the related data cluster comprising a set of training images and corresponding tags; and
means for propagating a tag to the image, the tag selected from the corresponding tags of the set of training images.

19. The system of claim 18 further comprising:
means for determining a relevance score for the tag; and
selecting the tag from the corresponding tags of the set of training images based on the relevance score.

Assignees

• Adobe Inc

Inventors

• Lin Zhe
• Shen Xiaohui
• Brandt Jonathan
• Zhang Jianming
• Fang Chen

Classifications

• G06N3/09

  Supervised learning · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06V20/35 Primary

  Categorising the entire scene, e.g. birthday party or wedding scene · CPC title

• G06F16/24578

  using ranking · CPC title

• G06F16/285

  Clustering or classification · CPC title