Method and apparatus for verifying accuracy of judgment result, electronic device and medium

Patent metadata

Field	Value
Publication number	US-12159235-B2
Application number	US-202117208485-A
Country	US
Kind code	B2
Filing date	Mar 22, 2021
Priority date	Sep 24, 2020
Publication date	Dec 3, 2024
Grant date	Dec 3, 2024

Abstract

A method and apparatus for verifying accuracy of a judgment result, an electronic device and a computer readable storage medium are provided. The method may include: acquiring user description information and a judgment result corresponding to the user description information; extracting at least one descriptive element from the user description information; determining a matching degree between each of the at least one descriptive element and the judgment result using a preset verification model to obtain each actual matching value respectively; the verification model being obtained by training based on a term frequency-inverse document frequency relationship between the descriptive element and the judgment result; and determining the descriptive element having the actual matching value exceeding a preset matching value as a first element, and determining accuracy of the judgment result based on the number of the first element.

First claim

What is claimed is:

1. A method for verifying accuracy of a judgment result, implemented by a server, the method comprising:
Acquiring, from a user terminal, user description information and a judgment result corresponding to the user description information, wherein the user description information comprises: a meeting log, content of a webpage article, or medical record information, and correspondingly the judgment result corresponding to the user description information comprises: whether the meeting log is written in a preset format, whether the content of the webpage article contains a sensitive word, or a suspected disease;
extracting at least one descriptive element from the user description information;
determining a matching degree between each of the at least one descriptive element and the judgment result using a preset verification model to obtain each actual matching value respectively; the verification model being obtained by training a neural network based on a term frequency-inverse document frequency TF-IDF relationship between the descriptive element and the judgment result;
determining, in the at least one descriptive element, a descriptive element having the actual matching value exceeding a preset matching value as a first element;
determining a plurality of knowledge elements related to the judgment result using a preset knowledge graph;
determining, in the at least one descriptive element, a descriptive element that is consistent with any of the knowledge elements as a second element; and
determining the accuracy of the judgment result based on the number of the first element and a number of the second element, highlighting the first element and the secodn element and displaying the highlighted first and second elements on the user terminal.

2. The method according to claim 1 , wherein, the determining the descriptive element that is consistent with any of the knowledge elements as the second element, comprises:
acquiring a confidence degree of each of the knowledge elements;
calculating to obtain a similarity between each of the at least one descriptive element and each of the knowledge elements respectively; and
determining the knowledge element and the descriptive element that are consistent based on the similarity and the confidence degree, and determining the descriptive element that is consistent with any of the knowledge elements as the second element.

3. The method according to claim 1 , wherein, the determining the accuracy of the judgment result based on the number of the first element and the number of the second element, comprises:
deduplicating the first element and the second element to obtain an element set; and
using a ratio of a number of elements in the element set to a total number of the knowledge elements as the accuracy of the judgment result.

4. The method according to claim 1 , further comprising:
adding the first element different from any of the knowledge elements to the knowledge graph as a supplementary element, in response to the first element being different from any of the knowledge elements.

5. The method according to claim 1 , wherein, the determining the accuracy of the judgment result based on the number of the first element and the number of the second element, comprises:
acquiring a first number weight and a second number weight set respectively for the first element and the second element in advance;
calculating the number of the first element and the first number weight, the number of the second element and the second number weight according to a weighted calculation method to obtain a weighted element number; and
determining the accuracy of the judgment result based on the weighted element number.

6. An electronic device, comprising:
at least one processor; and
a memory communicatively connected with the at least one processor;
the memory storing instructions executable by the at least one processor, and the instructions, when executed by the at least one processor, causing the at least one processor to perform operations, the operations comprising:
acquiring, from a user terminal, user description information and a judgment result corresponding to the user description information, wherein the user description information comprises: a meeting log, content of a webpage article, or medical record information, and correspondingly the judgment result corresponding to the user description information comprises: whether the meeting log is written in a preset format, whether the content of the webpage article contains a sensitive word, or a suspected disease;
extracting at least one descriptive element from the user description information;
determining a matching degree between each of the at least one descriptive element and the judgment result using a preset verification model to obtain each actual matching value respectively; the verification model being obtained by training a neural network based on a term frequency-inverse document frequency TF-IDF relationship between the descriptive element and the judgment result;
determining, in the at least one descriptive element, a descriptive element having the actual matching value exceeding a preset matching value as a first element;
determining a plurality of knowledge elements related to the judgment result using a preset knowledge graph;
determining, in the at least one descriptive element, a descriptive element that is consistent with any of the knowledge elements as a second element; and
determining the accuracy of the judgment result based on the number of the first element and a number of the second element, highlighting the first element and the second element and displaying the highlighted first and second elements on the user terminal.

7. The electronic device according to claim 6 , wherein, the determining the descriptive element that is consistent with any of the knowledge elements as the second element, comprises:
acquiring a confidence degree of each of the knowledge elements;
calculating to obtain a similarity between each of the at least one descriptive element and each of the knowledge elements respectively; and
determining the knowledge element and the descriptive element that are consistent based on the similarity and the confidence degree, and determining the descriptive element that is consistent with any of the knowledge elements as the second element.

8. The electronic device according to claim 6 , wherein, the determining the accuracy of the judgment result based on the number of the first element and the number of the second element, comprises:
deduplicating the first element and the second element to obtain an element set; and
using a ratio of a number of elements in the element set to a total number of the knowledge elements as the accuracy of the judgment result.

9. The electronic device according to claim 6 , the operations further comprising:
adding the first element different from any of the knowledge elements to the knowledge graph as a supplementary element, in response to the first element being different from any of the knowledge elements.

10. The electronic device according to claim 6 , wherein, the determining the accuracy of the judgment result based on the number of the first element and the number of the second element, comprises:
acquiring a first number weight and a second number weight set respectively for the first element and the second element in advance;
calculating the number of the first element and the first number weight, the number of the second element and the second number weight according to a weighted calculation method to obtain a weighted element number; and
determining the accuracy of the judgment result based on

Assignees

• Beijing Baidu Netcom Sci & Tech Co Ltd

Inventors

• Xing Zhihui
• Chen Jun
• Lu Chao
• Huang Haifeng

Classifications

• G06F16/9024

  Graphs; Linked lists (G06F16/9027 takes precedence) · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06N3/044

  Recurrent networks, e.g. Hopfield networks · CPC title

• G06N5/022

  Knowledge engineering; Knowledge acquisition · CPC title

• G06N3/08

  Learning methods · CPC title