Method and apparatus for establishing image set for image recognition, network device, and storage medium

Patent metadata

Field	Value
Publication number	US-11853352-B2
Application number	US-202017073051-A
Country	US
Kind code	B2
Filing date	Oct 16, 2020
Priority date	Oct 10, 2018
Publication date	Dec 26, 2023
Grant date	Dec 26, 2023

Abstract

A method of establishing an image set for image recognition includes obtaining a single-label image set comprising an image annotated with a single label, and a multi-label image set comprising an image annotated with a plurality of labels; converting content of each label into a corresponding word identifier according to a semantic network, to obtain a word identifier set, a converted single-label image set, and a converted multi-label image set; and constructing a hierarchical semantic structure according to the word identifier set and the semantic network. The method also includes performing label supplementation on the image in the converted single-label image set to obtain a supplemented single-label image set; performing label supplementation on the supplemented single-label image set to obtain a final supplemented image set; and establishing a target multi-label image set to train an image recognition model by using the target multi-label image set.

First claim

What is claimed is:

1. A method of establishing an image set for image recognition, performed by a network device, comprising:
obtaining a single-label image set and a multi-label image set, the single-label image set comprising an image annotated with a single label, and the multi-label image set comprising an image annotated with a plurality of labels;
converting content of each label into a corresponding word identifier according to a semantic network, to obtain a word identifier set, a converted single-label image set, and a converted multi-label image set;
constructing a hierarchical semantic structure according to the word identifier set and the semantic network, the hierarchical semantic structure comprising a semantic relationship between word identifiers;
performing label supplementation on the image in the converted single-label image set according to the semantic relationship between word identifiers, to obtain a supplemented single-label image set;
performing label supplementation on the image in the supplemented single-label image set based on a co-occurrence probability of a word identifier in the supplemented single-label image set and a word identifier in the converted multi-label image set, to obtain a final supplemented image set; and
establishing a target multi-label image set according to the final supplemented image set and the converted multi-label image set, to train an image recognition model by using the target multi-label image set.

2. The method according to claim 1 , wherein the performing label supplementation on the image in the converted single-label image set according to the semantic relationship between word identifiers, to obtain a supplemented single-label image set comprises:
obtaining a word identifier of a single-label image in the converted single-label image set, and querying the hierarchical semantic structure for an associated word identifier having a semantic relationship with the word identifier; and
supplementing and annotating the associated word identifier as a label of the single-label image, to obtain the supplemented single-label image set.

3. The method according to claim 1 , wherein the converting content of each label into a corresponding word identifier according to a semantic network, to obtain a word identifier set, a converted single-label image set, and a converted multi-label image set comprises:
converting the content of each label into the corresponding word identifier according to the semantic network, and merging and deduplicating same word identifiers, to obtain the word identifier set, the converted single-label image set, and the converted multi-label image set.

4. The method according to claim 3 , wherein the converting the content of each label into the corresponding word identifier according to the semantic network comprises:
converting the content of each label into the word identifier according to the semantic network, to obtain a plurality of candidate word identifiers corresponding to the label; and
selecting a word identifier corresponding to the image from the plurality of candidate word identifiers according to an image corresponding to the label, as the corresponding word identifier.

5. The method according to claim 1 , wherein before the converting content of each label into a corresponding word identifier according to a semantic network, the method further comprises:
obtaining a number of images corresponding to the label in the multi-label image set; and
deleting the label in a case that the number of images is less than a preset quantity.

6. The method according to claim 1 , wherein the performing label supplementation on the image in the supplemented single-label image set based on a co-occurrence probability of a word identifier in the supplemented single-label image set and a word identifier in the converted multi-label image set comprises:
obtaining a co-occurrence probability of a first-type word identifier and a second-type word identifier, the first-type word identifier being a word identifier in the supplemented single-label image set, the second-type word identifier being a word identifier in the converted multi-label image set;
determining, in the second-type word identifiers according to the co-occurrence probability, a target word identifier having a strong co-occurrence relationship with the first-type word identifier, the strong co-occurrence relationship comprising that the co-occurrence probability is greater than a preset probability and there is no semantic relationship between the two word identifiers; and
supplementing the target word identifier as a label of an image corresponding to the first-type word identifier in the supplemented single-label image set.

7. The method according to claim 6 , wherein the obtaining a co-occurrence probability of a first-type word identifier and a second-type word identifier comprises:
classifying, by using a trained classification model, an image with a label to be supplemented in the supplemented single-label image set, to obtain a predicted word identifier of the image with a label to be supplemented and a predicted probability of the predicted word identifier, the trained classification model being trained by using the converted multi-label image set;
determining, from the predicted word identifiers according to the predicted probabilities of the predicted word identifiers, a candidate supplementary label of the image with a label to be supplemented, to obtain an image set with the label determined, each image in the image set with the label determined carrying a built-in label and the candidate supplementary label, the built-in label being the word identifier corresponding to the first-type word identifier; and
obtaining the co-occurrence probability of the first-type word identifier and the second-type word identifier according to the image set with the label determined.

8. The method according to claim 7 , wherein the obtaining the co-occurrence probability of the first-type word identifier and the second-type word identifier according to the image set with the label determined comprises:
counting, in the image set with the label determined, a number of images on which both the first-type word identifier and the second-type word identifier appear and a total number of images on which the first-type word identifier appears; and
obtaining the cooccurrence probability of the first-type word identifier and the second-type word identifier according to the number of images and the total number of images.

9. The method according to claim 6 , wherein the determining, in the second-type word identifiers according to the co-occurrence probability, a target word identifier having a strong co-occurrence relationship with the first-type word identifier comprises:
determining, in the second-type word identifiers, the target word identifier having a strong co-occurrence relationship with the first-type word identifier according to the co-occurrence probability and a semantic relationship between the first-type word identifier and the second-type word identifier.

10. A network device, comprising:
a memory storing computer program instructions; and
a processor coupled to the memory and, when executing the computer program instructions, configured to perform:
obtaining a single-label image set and a multi-label image set, the single-label image set comprising an image annotated with a single label, and the multi-label image set comprising an image annotated with a plurality of labels;
converting content of each label into a corresponding word identifier according to a semantic network, to obtain a word identifier set, a converted single-label image set, and a converted

Assignees

• Tencent Tech Shenzhen Co Ltd

Inventors

• Wu Baoyuan
• Chen Weidong
• Liu Wei
• Fan Yanbo
• Zhang Tong

Classifications

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06F16/58 Primary

  Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually · CPC title

• G06F16/535

  Filtering based on additional data, e.g. user or group profiles · CPC title

• G06F16/55

  Clustering; Classification · CPC title