Automated citations and assessment for automatically generated text

Patent metadata

Field	Value
Publication number	US-12423509-B2
Application number	US-202318345508-A
Country	US
Kind code	B2
Filing date	Jun 30, 2023
Priority date	Jun 30, 2023
Publication date	Sep 23, 2025
Grant date	Sep 23, 2025

Abstract

In some implementations, the techniques described herein relate to a method including: parsing, by a processor, a generated text to identify statements included within a generated text; querying, by the processor, a remote data source to identify sources for each statement in the statements; determining, by the processor, trustworthiness values for each statement, a trustworthiness value for a given statement determined by computing trustworthiness labels for each source corresponding to a given statement: generating, by the processor, a label for the generated text based on an aggregated trustworthiness of each of the statements; and displaying, by the processor, the generated text and the label within a user interface displayed to a user.

First claim

We claim:

1. A method comprising:
parsing, by a processor, a generated text to identify statements included within a generated text;
querying, by the processor, a remote data source to identify sources for each statement in the statements;
determining, by the processor, trustworthiness values for each statement, a trustworthiness value for a given statement determined by computing trustworthiness labels for each source corresponding to a given statement:
generating, by the processor, a label for the generated text based on an aggregated trustworthiness of each of the statements;
displaying, by the processor, the generated text and the label within a user interface displayed to a user;
displaying statement labels for each of the statements, the statement labels representing trustworthiness of each of the statements; and
detecting a selection of a given statement label and displaying a corresponding list of sources.

2. The method of claim 1 , further comprising generating the generated text by inserting a prompt into a large language model (LLM).

3. The method of claim 1 , wherein identifying the statements included within the generated text includes:
parsing the generated text into a set of sentences;
filtering the set of sentences to identify a second set of sentences that includes informational content; and
using the second set of sentences as the statements.

4. The method of claim 3 , wherein identifying the statements included within the generated text includes performing co-reference resolution on the generated text prior to parsing the generated text.

5. The method of claim 1 , wherein identifying a given source in the sources comprises using a corresponding statement as a search query and querying a search engine to obtain a list of search results, the list of search results comprising the sources.

6. The method of claim 1 , wherein determining trustworthiness values of each statement comprises:
classifying a trustworthiness of each source in sources corresponding to the given statement;
scoring and sorting the sources corresponding to the given statement;
selecting a top subset of the sources; and
determining a given trustworthiness value of the given statement based on classified trustworthiness values of each source in the top subset of the sources.

7. The method of claim 6 , wherein selecting a top subset of the sources comprises performing an entailment analysis using the given statement and a respective source.

8. A non-transitory computer-readable storage medium for tangibly storing computer program instructions capable of being executed by a computer processor, the computer program instructions defining steps of:
parsing a generated text to identify statements included within a generated text;
querying a remote data source to identify sources for each statement in the statements;
determining trustworthiness values for each statement, a trustworthiness value for a given statement determined by computing trustworthiness labels for each source corresponding to a given statement:
generating a label for the generated text based on an aggregated trustworthiness of each of the statements;
displaying the generated text and the label within a user interface displayed to a user;
displaying statement labels for each of the statements, the statement labels representing trustworthiness of each of the statements; and
detecting a selection of a given statement label and displaying a corresponding list of sources.

9. The non-transitory computer-readable storage medium of claim 8 , the steps further comprising generating the generated text by inserting a prompt into a large language model (LLM).

10. The non-transitory computer-readable storage medium of claim 8 , wherein identifying the statements included within the generated text includes:
parsing the generated text into a set of sentences;
filtering the set of sentences to identify a second set of sentences that includes informational content; and
using the second set of sentences as the statements.

11. The non-transitory computer-readable storage medium of claim 10 , wherein identifying the statements included within the generated text includes performing co-reference resolution on the generated text prior to parsing the generated text.

12. The non-transitory computer-readable storage medium of claim 8 , wherein identifying a given source in the sources comprises using a corresponding statement as a search query and querying a search engine to obtain a list of search results, the list of search results comprising the sources.

13. The non-transitory computer-readable storage medium of claim 8 , wherein determining trustworthiness values of each statement comprises:
classifying a trustworthiness of each source in sources corresponding to the given statement;
scoring and sorting the sources corresponding to the given statement;
selecting a top subset of the sources; and
determining a given trustworthiness value of the given statement based on classified trustworthiness values of each source in the top subset of the sources.

14. The non-transitory computer-readable storage medium of claim 13 , wherein selecting a top subset of the sources comprises performing an entailment analysis using the given statement and a respective source.

15. A device comprising:
a processor; and
a storage medium for tangibly storing thereon logic for execution by the processor, the logic comprising instructions for:
parsing a generated text to identify statements included within a generated text;
querying a remote data source to identify sources for each statement in the statements;
determining trustworthiness values for each statement, a trustworthiness value for a given statement determined by computing trustworthiness labels for each source corresponding to a given statement:
generating a label for the generated text based on an aggregated trustworthiness of each of the statements;
displaying the generated text and the label within a user interface displayed to a user;
displaying statement labels for each of the statements, the statement labels representing trustworthiness of each of the statements; and
detecting a selection of a given statement label and displaying a corresponding list of sources.

16. The device of claim 15 , wherein identifying the statements included within the generated text includes:
parsing the generated text into a set of sentences;
filtering the set of sentences to identify a second set of sentences that includes informational content; and
using the second set of sentences as the statements.

17. The device of claim 15 , wherein identifying a given source in the sources comprises using a corresponding statement as a search query and querying a search engine to obtain a list of search results, the list of search results comprising the sources.

18. The device of claim 15 , wherein determining trustworthiness values of each statement comprises:
classifying a trustworthiness of each source in sources corresponding to the given statement;
scoring and sorting the sources corresponding to the given statement;
selecting a top subset of the sources by performing an entailment analysis using the given statement and a respective source; and
determining a given trustworthiness value of the given statement based on classified trustworthiness values of each source in the top subset of the sources.

Assignees

• Yahoo Assets Llc

Inventors

• Bax Eric
• Sarkar Arundhyoti
• Gerber Melissa
• Wagner Will
• Giaffo Lisa
• Thompson Nikki

Classifications

• G06F40/205

  Parsing · CPC title

• G06F16/382 Primary

  using citations (hypermedia G06F16/94) · CPC title

• G06F16/35

  Clustering; Classification · CPC title

• G06F40/169 Primary

  Annotation, e.g. comment data or footnotes · CPC title