Method, device, computer equipment and storage medium for identifying illegal commodity

Patent metadata

Field	Value
Publication number	US-12450933-B2
Application number	US-202318460680-A
Country	US
Kind code	B2
Filing date	Sep 4, 2023
Priority date	Apr 3, 2023
Publication date	Oct 21, 2025
Grant date	Oct 21, 2025

Abstract

A method, a device, computer equipment and a storage medium for identify an illegal commodity. The method comprises: firstly, constructing a multi-modal knowledge graph according to a multi-modal knowledge graph data set, and extracting visual features of all visual modality entities and text features of all text modality entities in the knowledge graph; then obtaining a commodity image and a commodity text according to a database; then, generating commodity visual feature according to the commodity image; then generating the commodity text feature according to the commodity text; secondly, according to the visual features and text features, as well as the commodity visual feature and the commodity text feature, linking the commodity image and the commodity text to the knowledge graph by using an entity linking method; finally, obtaining the correlation between the commodity image and the commodity text according to the linked knowledge graph to determine the illegality of the commodity.

First claim

What is claimed is:

1. A method for identifying an illegal commodity for monitoring e-commerce platforms, comprising:
step (1) constructing a multi-modal knowledge graph based on a multi-modal knowledge graph data set, and extracting visual features of all visual modality entities and text features of a text modality entities in the multi-modal knowledge graph;
step (2) acquiring commodity information based on a database, wherein the commodity information comprises a commodity image and a commodity text, and the commodity text comprises at least one of a commodity title and a commodity description;
step (3) generating a commodity visual feature based on the commodity image;
step (4) generating a commodity text feature based on the commodity text;
step (5) linking the commodity image and the commodity text to the multi-modal knowledge graph constructed in the step (1) using an entity linking method, based on the visual features and the text features extracted in the step (1) as well as the commodity visual feature generated in the step (3) and the commodity text feature generated in the step (4); and
step (6) acquiring a correlation between the commodity image and the commodity text based on the linked multi-modal knowledge graph obtained in the step (5) to determine illegality of a commodity by a discriminator.

2. The method for identifying the illegal commodity according to claim 1 , wherein the multi-modal knowledge graph data set comprises Wikidata, DBpedia, YAGO, Concept and WordNet.

3. The method for identifying the illegal commodity according to claim 1 , wherein said acquiring the correlation between the commodity image and the commodity text based on the linked multi-modal knowledge graph obtained in the step (5) to determine the illegality of the commodity comprises: acquiring a shortest path from the commodity image to the commodity text in the linked multi-modal knowledge graph, and determining the illegality of the commodity based on the shortest path.

4. The method for identifying the illegal commodity according to claim 3 , wherein said determining the illegality of the commodity based on the shortest path comprises: acquiring a length of the shortest path based on the shortest path, comparing the length of the shortest path with a set length threshold, and when the length of the shortest path is greater than the length threshold, determining that the commodity is illegal; and otherwise, determining that the commodity is not illegal.

5. The method for identifying the illegal commodity according to claim 3 , wherein said determining the illegality of the commodity based on the shortest path comprises: acquiring neighbor nodes of all nodes in the shortest path within k hops and edges for connecting the neighbor nodes, and forming a subgraph together with the shortest path; acquiring features of nodes and edges in the subgraph; constructing a discriminative model and training the discriminative model to acquire a trained discriminative model; and inputting the subgraph and the features of the nodes and the edges in the subgraph into the trained discriminative model for discrimination, so as to acquire the illegality of the commodity.

6. The method for identifying the illegal commodity according to claim 5 , wherein the discriminative model comprises:
a graph convolution neural network with k layers or more configured to aggregate a k-hop neighbor information of the nodes in the shortest path to the shortest path;
a sequence model configured to process the features of the nodes and the edges in the shortest path after the k-hop neighbor information is aggregated, so as to acquire correlation features of the nodes in the shortest path; and
a classifier configured to process sequence correlation features of the shortest path and provide a determination result of the illegality of the commodity.

7. The method for identifying the illegal commodity according to claim 5 , wherein said training the discriminative model to acquire the trained discriminative model comprises:
sub-step (a1) collecting a commodity data based on a database to construct a training data set, wherein the training data set comprises a plurality of instances, and each of the instance comprises a commodity image, a commodity text and a label indicating whether the commodity is illegal of a same commodity;
sub-step (a2) randomly sampling a batch of instances from the training data set, linking the commodity image and the commodity text in every instance to a multi-modal knowledge graph, respectively, to acquire a batch of subgraphs, and inputting the subgraphs and the features of the nodes and the edges in the subgraphs in the instances into the constructed discriminative model to acquire a prediction result;
sub-step (a3) calculating a loss based on the prediction result and the label indicating whether the commodity of the instances in the training data set is illegal;
sub-step (a4) updating parameters of the discriminative model by a back propagation method and a gradient descent method based on the loss calculated in the sub-step (a3); and
sub-step (a5) repeating the sub-step (a2) to the sub-step (a4) until the loss converges to acquire the trained discriminative model.

8. A device for identifying an illegal commodity applied to implementing the method for identifying the illegal commodity according to claim 1 , wherein the device comprises:
a commodity information acquisition module configured to acquire commodity information, wherein the commodity information comprises a commodity image and a commodity text, and the commodity text comprises at least one of a commodity title and a commodity description;
an image feature extraction module configured to generate a commodity visual feature based on the commodity image;
a text feature extraction module configured to generate a commodity text feature based on the commodity text;
a linking module configured to link the commodity image and the commodity text to a multi-modal knowledge graph based on the commodity visual feature and the commodity text feature; and
a determination module configured to determine the illegality of the commodity based on the linked multi-modal knowledge graph.

9. A computer equipment, comprising a memory and a processor, wherein the memory is configured to store program data, and the processor is configured to executing the program data to implement the method for identifying the illegal commodity according to claim 1 .

10. A computer-readable storage medium on which a computer program is stored, wherein the program, when executed by a processor, is configured to implement the method for identifying the illegal commodity according to claim 1 .

Assignees

• Zhejiang Lab

Inventors

• Qi Yao
• Chen Hongyang
• Lv Jingsong
• Yang Wentao

Classifications

• G06N5/02

  Knowledge representation; Symbolic representation · CPC title

• G06V30/19173 Primary

  Classification techniques · CPC title

• G06N3/08

  Learning methods · CPC title

• G06V10/82

  using neural networks · CPC title

• G06N3/045

  Combinations of networks · CPC title