Systems and methods to identify and distribute diverse evaluation assignments

Patent metadata

Field	Value
Publication number	US-2026010858-A1
Application number	US-202418762846-A
Country	US
Kind code	A1
Filing date	Jul 3, 2024
Priority date	Jul 3, 2024
Publication date	Jan 8, 2026
Grant date	—

Abstract

Interaction distribution systems and methods, and non-transitory computer readable media, include retrieving a diverse evaluation configuration including diverse evaluation configuration rules and diverse interaction category rules; retrieving historical evaluations for each evaluator based on the diverse evaluation configuration rules; retrieving diverse criteria prompt rules; retrieving an interaction transcript associated with each historical evaluation; constructing a large language model (LLM) prompt based on the diverse criteria prompt rules; executing the first LLM prompt on each interaction transcript to return a category of interaction for each interaction transcript; determining evaluation coverage for each returned category of interaction for each evaluator; determining that an evaluator has not evaluated a defined number of interactions for one or more of the diverse interaction category rules; and distributing one or more interactions that match the one or more diverse interaction category rules to the evaluator for evaluation.

First claim

What is claimed is:

1 . An interaction distribution system comprising:
a processor and a non-transitory computer readable medium operably coupled thereto, the non-transitory computer readable medium comprising a plurality of instructions stored in association therewith that are accessible to, and executable by, the processor, to perform operations which comprise:
retrieving a diverse evaluation configuration, wherein the diverse evaluation configuration comprises diverse evaluation configuration rules for a plurality of evaluators and the diverse evaluation configuration rules comprise diverse interaction category rules;
retrieving historical evaluations for each evaluator from the plurality of evaluators based on the diverse evaluation configuration rules;
retrieving diverse criteria prompt rules;
retrieving an interaction transcript associated with each historical evaluation;
constructing a first large language model (LLM) prompt based on the diverse criteria prompt rules;
executing the first LLM prompt on each interaction transcript to return a category of interaction for each interaction transcript;
determining evaluation coverage for each returned category of interaction for each evaluator;
determining that an evaluator has not evaluated a defined number of interactions for one or more of the diverse interaction category rules; and
distributing one or more interactions that match the one or more diverse interaction category rules to the evaluator for evaluation.

2 . The interaction distribution system of claim 1 , wherein the diverse evaluation configuration rules further comprise one or more interaction selection rules, names of evaluators, and an evaluation assignment schedule.

3 . The interaction distribution system of claim 1 , wherein determining evaluation coverage for each returned category of interaction for each evaluator comprises calculating a diverse assignment of evaluation (DAE) score for each returned category of interaction for each evaluator.

4 . The interaction distribution system of claim 3 , wherein the operations further comprise:
generating a coverage report that includes the calculated DAE score; and displaying the coverage report to a manager of the evaluator.

5 . The interaction distribution system of claim 4 , wherein determining that an evaluator has not evaluated a defined number of interactions for one or more of the diverse interaction category rules comprises reviewing the coverage report.

6 . The interaction distribution system of claim 1 , wherein distributing one or more interactions that match the one or more diverse interaction category rules to the evaluator for evaluation comprises:
retrieving a plurality of interactions for the evaluator; analyzing the retrieved plurality of interactions to ensure the retrieved plurality of interactions match the one or more diverse interaction category rules; and assigning the analyzed, retrieved plurality of interactions to the evaluator.

7 . The interaction distribution system of claim 6 , wherein retrieving a plurality of interactions for the evaluator comprises:
determining a defined number of unique interactions from the diverse evaluation configuration rules; applying a sampling factor to the defined number of unique interactions; and sampling interactions from an interaction database based on the diverse evaluation configuration rules.

8 . The interaction distribution system of claim 7 , wherein analyzing the retrieved plurality of interactions to ensure the retrieved interactions match the one or more diverse interaction category rules comprises:
constructing a second LLM prompt based on the diverse criteria prompt rules; executing the second LLM prompt on each sampled interaction to return a category of interaction for each sampled interaction; and filtering out the sampled interactions that match the one or more diverse interaction category rules.

9 . The interaction distribution system of claim 8 , wherein assigning the analyzed, retrieved plurality of interactions to the evaluator for evaluation comprises
mapping the filtered, sampled interactions to the evaluator based on the coverage report; and creating evaluation tasks for the evaluator.

10 . A method for distributing interactions for evaluation, which comprises:
retrieving a diverse evaluation configuration, wherein the diverse evaluation configuration comprises diverse evaluation configuration rules for a plurality of evaluators and the diverse evaluation configuration rules comprise diverse interaction category rules; retrieving historical evaluations for each evaluator from the plurality of evaluators based on the diverse evaluation configuration rules; retrieving diverse criteria prompt rules; retrieving an interaction transcript associated with each historical evaluation; constructing a first large language model (LLM) prompt based on the diverse criteria prompt rules; executing the first LLM prompt on each interaction transcript to return a category of interaction for each interaction transcript; determining evaluation coverage for each returned category of interaction for each evaluator; determining that an evaluator has not evaluated a defined number of interactions for one or more of the diverse interaction category rules; and distributing one or more interactions that match the one or more diverse interaction category rules to the evaluator for evaluation.

11 . The method of claim 10 , wherein determining evaluation coverage for each returned category of interaction for each evaluator comprises calculating a diverse assignment of evaluation (DAE) score for each returned category of interaction for each evaluator.

12 . The method of claim 11 , which further comprises:
generating a coverage report that includes the calculated DAE score; and displaying the coverage report to a manager of the evaluator.

13 . The method of claim 10 , wherein distributing one or more interactions that match the one or more diverse interaction category rules to the evaluator for evaluation comprises:
retrieving a plurality of interactions for the evaluator; analyzing the retrieved plurality of interactions to ensure the retrieved plurality of interactions match the one or more diverse interaction category rules; and assigning the analyzed, retrieved plurality of interactions to the evaluator.

14 . The method of claim 13 , wherein retrieving a plurality of interactions for the evaluator comprises:
determining a defined number of unique interactions from the diverse evaluation configuration rules; applying a sampling factor to the defined number of unique interactions; and sampling interactions from an interaction database based on the diverse evaluation configuration rules.

15 . The method of claim 14 , wherein analyzing the retrieved plurality of interactions to ensure the retrieved interactions match the one or more diverse interaction category rules comprises:
constructing a second LLM prompt based on the diverse criteria prompt rules; executing the second LLM prompt on each sampled interaction to return a category of interaction for each sampled interaction; and filtering out the sampled interactions that match the one or more diverse interaction category rules.

16 . A non-transitory computer-readable medium having stored thereon computer-readable instructions executable by a processor to perform operations which comprise:
building a library comprising previously identified stressful sentences and stressful phrases;
retrieving a diverse evaluation configuration, wh

Assignees

• Nice Ltd

Inventors

• Vyas Rahul
• Hopkins Leann
• Lamba Devinder
• Mokashi Abhijit

Classifications

• G06F40/51 Primary

  Translation evaluation · CPC title

• G06Q10/06398 Primary

  Performance of employee with respect to a job function · CPC title