Precision evaluation system for artificial intelligence generated text summaries

Patent metadata

Field	Value
Publication number	US-2025272514-A1
Application number	US-202418584720-A
Country	US
Kind code	A1
Filing date	Feb 22, 2024
Priority date	Feb 22, 2024
Publication date	Aug 28, 2025
Grant date	—

Abstract

A machine learning (ML) system and methods are provided that are configured to evaluate artificial intelligence (AI) generated text summaries using an evaluation framework for a weighting strategy of a plurality of metrics. The ML system includes a processor and a computer readable medium operably coupled thereto, the computer readable medium comprising a plurality of instructions stored in association therewith that are accessible to, and executable by, the processor, to perform summary evaluation operations which include accessing a AI generated text summary and an original text, calculating a plurality of summarization evaluation metrics, weighting the plurality of summarization evaluation metrics, computing a final evaluation score based on an aggregation of the weighted plurality of summarization evaluation metrics, outputting the precision evaluation based on the computed final evaluation score, and updating a data structure for the texts with the computed final evaluation score.

First claim

What is claimed is:

1 . A machine learning (ML) system configured to evaluate artificial intelligence (AI) generated text summaries using an evaluation framework for a weighting strategy of a plurality of metrics, the ML system comprising:
a processor and a computer readable medium operably coupled thereto, the computer readable medium comprising a plurality of instructions stored in association therewith that are accessible to, and executable by, the processor, to perform summary evaluation operations which comprise:
accessing, for a precision evaluation of an AI generated text summary using the evaluation framework, a data structure including the AI generated text summary and an original text summarized in the AI generated text summary by an AI summarization engine;
calculating a plurality of summarization evaluation metrics that evaluate a summarization of the original text in the AI generated text summary using the evaluation framework with the AI generated text summary and the original text, wherein the plurality of summarization evaluation metrics are selected for the evaluation framework based on a relevance assessment and a significance assessment of each of the plurality of summarization evaluation metrics when performing the precision evaluation of the summarization;
weighting the plurality of summarization evaluation metrics based on a plurality of weights applied by the evaluation framework;
computing a final evaluation score of the summarization of the original text in the AI generated text summary by the AI summarization engine based on an aggregation of the weighted plurality of summarization evaluation metrics;
outputting the precision evaluation of the AI generated text summary based on the computed final evaluation score; and
updating the data structure with the computed final evaluation score for the precision evaluation.

2 . The ML system of claim 1 , wherein one of the plurality of summarization evaluation metrics comprises a global coherence metric configured to assess a topic similarity between the original text and the AI generated text summary.

3 . The ML system of claim 2 , wherein the calculating the plurality of summarization evaluation metrics comprises calculating the global coherence metric by:
preprocessing the original text and the AI generated text summary; creating a text corpus representing the original text and the AI generated text summary for a topic modeling of topics used for assessing the topic similarity; training a Latent Dirichlet Allocation (LDA) model using the text corpus, wherein the LDA model assigns the topics to each of the original text and the AI generated text summary based on a distribution of words in the original text and the AI generated text summary; obtaining a topic distribution of the topics in each of the original text and the AI generated text summary using the LDA model with the original text and the AI generated text summary; comparing the topic distribution of the topics in the original text with the topic distribution of the topics in the AI generated text summary; and returning a similarity score of the topics between the original text and the AI generated text summary based on the comparing.

4 . The ML system of claim 3 , wherein the calculating the global coherence metric utilizes at least one of a preprocessing string function, a topic modeling library for the LDA model, a Hellinger distance for a similarity measurement corresponding to the similarity score, or a similarity score calculation with the similarity measure for the similarity score.

5 . The ML system of claim 2 , wherein the plurality of summarization evaluation metrics further comprise at least one of a recall-oriented understudy for Gisting evaluation (ROUGE)-2 score, a ROUGE-L score, a bilingual evaluation understudy (BLEU) score, a metric for evaluation of translation with explicit ordering (METEOR) score, a bidirectional encoder representations from transformers (BERT) score, an entity preservation score, a semantic similarity score using BERT or robustly optimized BERT pretraining approach (ROBERTa), a length ratio, a precision-at-K score, a word error rate, a normalized cross entropy, or an overlap coefficient.

6 . The ML system of claim 1 , wherein the plurality of summarization evaluation metrics are split into four categories each having a subset of the plurality of summarization evaluation metrics, and wherein the plurality of weights comprise one of a same weight or a different configurable weight applied to each of the four categories when performing the weighting the plurality of summarization evaluation metrics.

7 . The ML system of claim 6 , wherein the four categories comprise a content quality metrics category, a coherence and structure metrics category, a semantic similarity metrics category, and an entity preservation metrics category.

8 . The ML system of claim 7 , wherein calculating the plurality of summarization evaluation metrics include separately calculating each of the four categories using the subset of the plurality of summarization evaluation metrics for a corresponding one of the four categories.

9 . The ML system of claim 1 , wherein, prior to the calculating the plurality of summarization evaluation metrics, the summary evaluation operations further comprise:
preprocessing the AI generated text summary and the original text; and tokenizing preprocessed text in the AI generated text summary and the original text, wherein the calculating the plurality of summarization evaluation metrics using the evaluation framework is with the tokenized preprocessed text from the AI generated text summary and the original text.

10 . A method to evaluate artificial intelligence (AI) generated text summaries using an evaluation framework for a weighting strategy of a plurality of metrics by a machine learning (ML) system, the method comprising:
accessing, for a precision evaluation of an AI generated text summary using the evaluation framework, a data structure including the AI generated text summary and an original text summarized in the AI generated text summary by an AI summarization engine; calculating a plurality of summarization evaluation metrics that evaluate a summarization of the original text in the AI generated text summary using the evaluation framework with the AI generated text summary and the original text, wherein the plurality of summarization evaluation metrics are selected for the evaluation framework based on a relevance assessment and a significance assessment of each of the plurality of summarization evaluation metrics when performing the precision evaluation of the summarization; weighting the plurality of summarization evaluation metrics based on a plurality of weights applied by the evaluation framework; computing a final evaluation score of the summarization of the original text in the AI generated text summary by the AI summarization engine based on an aggregation of the weighted plurality of summarization evaluation metrics; outputting the precision evaluation of the AI generated text summary based on the computed final evaluation score; and updating the data structure with the computed final evaluation score for the precision evaluation.

11 . The method of claim 10 , wherein one of the plurality of summarization evaluation metrics comprises a global coherence metric configured to assess a topic similarity between the original text and the AI generated text summary.

12 . The method of claim 11 , wherein the calculating the plurality of summarization evaluation metrics comprises calculating the global coherence metric by:
preprocessing the original text and the

Assignees

• Actimize Ltd

Inventors

• Butvinik Danny

Classifications

• G06F40/56

  Natural language generation · CPC title

• G06F40/51 Primary

  Translation evaluation · CPC title

• G06F40/30 Primary

  Semantic analysis · CPC title