Systems and methods for extracting evidence to facilitate claim verification

Patent metadata

Field	Value
Publication number	US-11720754-B2
Application number	US-202117160128-A
Country	US
Kind code	B2
Filing date	Jan 27, 2021
Priority date	Jan 27, 2021
Publication date	Aug 8, 2023
Grant date	Aug 8, 2023

Abstract

Claim verification is facilitated by identifying a selection of textual content within a user interface, accessing a claim detection module, inputting the textual content into the claim detection module to detect one or more claims within the selection of textual content, accessing an evidence extraction module, using the evidence extraction module to automatically search one or more reference repositories for one or more related references that have content that is related to the detected one or more claims, automatically determining whether the content in the one or more related references supports or refutes the one or more claims using a claim verification module, and, in response to determining a set of references of the one or more related references supports or refutes the one or more claims, presenting a support indicator or a refute indicator within the user interface in association with the one or more claims.

First claim

We claim:

1. A system for facilitating claim verification, comprising:
one or more processors; and
one or more hardware storage devices storing instructions that are executable by the one or more processors to configure the system to:
identify a selection of textual content within a user interface;
access a claim detection module that has been trained with a first set of ground truth to enable the claim detection module to detect occurrences of claims within text;
input the textual content into the claim detection module to detect one or more claims within the selection of textual content;
in response to detecting the one or more claims within the textual content, access an evidence extraction module that has been trained with a second set of ground truth to enable the evidence extraction module to identify references from one or more reference repositories that have content related to one or more particular types of claims;
further, in response to detecting the one or more claims within the textual content, use the evidence extraction module to automatically search the one or more reference repositories for one or more related references that have one or more pieces of evidence that are determined by the evidence extraction module to be related to the detected one or more claims;
in response to identifying the one or more related references, automatically determine whether the content in the one or more related references supports or refutes the one or more claims using a claim verification module, the claim verification module having been trained with a third set of ground truth to determine whether the one or more pieces of evidence support or refute the one or more claims;
in response to determining a set of references of the one or more related references supports or refutes the one or more claims, present at least one of (i) a support indicator or (ii) a refute indicator within the user interface in association with the one or more claims, the support indicator visually indicating that the one or more claims are supported by evidence, the refute indicator visually indicating that the one or more claims are refuted by evidence;
present a representation of the set of references in the user interface in association with the one or more claims;
present a representation of one or more pieces of evidence of the set of references in the user interface in association with the one or more claims;
receive user input indicating that a reference or a piece of evidence of the set of references is not relevant to the one or more claims; and
further train the evidence extraction module based on the user input indicating that the reference or the piece of evidence of the set of references is not relevant to the one or more claims.

2. The system of claim 1 , wherein the instructions are executable by the one or more processors to further configure the system to:
in response to determining the set of references of the one or more related references supports the one or more claims, present the support indicator within the user interface in association with the one or more claims.

3. The system of claim 1 , wherein the instructions are executable by the one or more processors to further configure the system to:
in response to determining the set of references of the one or more related references refutes the one or more claims, present the refute indicator within the user interface in association with the one or more claims.

4. The system of claim 1 , wherein the support indicator or the refute indicator indicates a weight of support or weight of refutation for the one or more claims.

5. The system of claim 1 , wherein the instructions are executable by the one or more processors to further configure the system to:
present a representation of the set of references in the user interface in association with the one or more claims.

6. The system of claim 5 , wherein the instructions are executable by the one or more processors to further configure the system to:
present a representation of one or more pieces of evidence of the set of references in the user interface in association with the one or more claims.

7. The system of claim 6 , wherein the instructions are executable by the one or more processors to further configure the system to:
receive user input indicating that a reference or a piece of evidence of the set of references is not relevant to the one or more claims; and
refine the evidence extraction module based on the user input indicating that the reference or the piece of evidence of the set of references is not relevant to the one or more claims.

8. The system of claim 6 , wherein the instructions are executable by the one or more processors to further configure the system to:
receive user input indicating that a reference or a piece of evidence of the set of references fails to support the one or more claims; and
refine the claim verification module based on the user input indicating that the reference or the piece of evidence of the set of references fails to support the one or more claims.

9. The system of claim 6 , wherein the instructions are executable by the one or more processors to further configure the system to:
receive user input indicating that a reference or a piece of evidence of the set of references fails to refute the one or more claims; and
refine the claim verification module based on the user input indicating that the reference or the piece of evidence of the set of references fails to refute the one or more claims.

10. The system of claim 1 , wherein the set of references omits a particular reference that fails to satisfy a classification confidence threshold, the classification confidence threshold indicating a level of confidence in a determination by the claim verification module that a reference supports or refutes a claim.

11. The system of claim 1 , wherein the instructions are executable by the one or more processors to further configure the system to:
present a recommendation for modifying the one or more claims based on the set of references.

12. The system of claim 1 , wherein the instructions are executable by the one or more processors to further configure the system to:
emphasize one or more portions of the textual content determined by the claim detection module to be contextually associated with the one or more claims;
receive user input modifying the one or more portions of the textual content that are contextually associated with the one or more claims;
redetermine the one or more claims using the claim detection module and based upon the user input modifying the one or more portions of the textual content; and
further train the claim detection module based on the user input modifying the one or more portions of the textual content that are contextually associated with the one or more claims.

13. The system of claim 1 , wherein the one or more related references identified using the evidence extraction module satisfy a threshold of relevance to content of detected claims.

14. A system for facilitating claim verification, comprising:
one or more processors; and
one or more hardware storage devices storing instructions that are executable by the one or more processors to configure the system to:
identify a selection of textual content within a user interface;
access a claim detection module that has been trained with a first set of ground truth to enable the claim detection module to detect occurrences of claims within text;
input the textual content into the claim detection module to detect one or more claims within the selection of textual content;
in respo

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Abraham Robin
• Du Liang
• Ahmed Zeeshan

Classifications

• G06F40/30 Primary

  Semantic analysis · CPC title

• G06F18/214

  Generating training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06F40/284

  Lexical analysis, e.g. tokenisation or collocates · CPC title

• G06F16/93 Primary

  Document management systems · CPC title

• G06F16/35

  Clustering; Classification · CPC title