System, method and computer program product for assessing the capabilities of a conversation agent via black box testing

Patent metadata

Field	Value
Publication number	US-10102846-B2
Application number	US-201615338695-A
Country	US
Kind code	B2
Filing date	Oct 31, 2016
Priority date	Oct 31, 2016
Publication date	Oct 16, 2018
Grant date	Oct 16, 2018

Abstract

A conversational agent capability assessment method, system, and computer program product, includes obtaining data to create at least one scenario for testing a conversational agent, performing a set of tests using a scenario of the at least one scenario created to assess a capability of the conversational agent, and comparing a result of the capability from the set of tests with an expected result of the scenario.

First claim

What is claimed is:

1. A computer-implemented conversational agent capability assessment method, the method comprising:
obtaining data to create at least one scenario for testing a conversational agent in a conversational system having a non-deterministic output, the conversational agent being a software based on natural language classifiers;
performing a set of tests using a scenario of the at least one scenario created to assess a capability of the conversational agent based on evaluation metrics generated from the set of tests while interacting with the conversational agent, the set of tests comprising testing a plurality of metrics including:
a performance of the conversational agent;
a linguistic variation;
a personality type of the conversation agent; and
a cognitive trait of the conversational agent; and

comparing a result of the capability from the set of tests with an expected result of the scenario based on results from a different conversational agent,
wherein the evaluation metrics are generated while interacting with the conversational system embodied in a cloud-computing environment.

2. The computer-implemented method of claim 1 , wherein each of the at least one created scenario comprises a user input to cause an aspect of the conversational agent to be tested.

3. The computer-implemented method of claim 1 , wherein the performing the set of tests comprises testing at least one of:
a natural language ability of the conversational agent;
pattern dialogue flow of the conversational agent;
a response to an unexpected user input by the conversational agent; and
a knowledge base of the conversational agent.

4. The computer-implemented method of claim 1 , wherein the expected result includes at least one of:
a capability of a different conversational agent; and
a capability of a different version of a same conversational agent being tested.

5. The computer-implemented method of claim 1 , wherein the at least one created scenario includes a speech-to-text and a text-to-speech conversion to assess the capability of the conversational agent for both of a textual input conversational agent and a speech input conversational agent.

6. The computer-implemented method of claim 1 , further comprising storing the result from the set of tests to be used for a future comparison by the comparing.

7. The computer-implemented method of claim 1 , further comprising generating a report of the result of the capability of the conversational agent for a human user.

8. The computer-implemented method of claim 7 , further comprising suggesting an action for the human user to modify the conversational agent based on the report of the result of the capability of the conversational agent.

9. The computer-implemented method of claim 1 , wherein the performing performs the set of tests by testing at least two of the plurality of testing metrics.

10. A computer program product for conversational agent capability assessment, the computer program product comprising a computer-readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to perform:
obtaining data to create at least one scenario for testing a conversational agent in a conversational system having a non-deterministic output, the conversational agent being a software based on natural language classifiers;
performing a set of tests using a scenario of the at least one scenario created to assess a capability of the conversational agent based on evaluation metrics generated from the set of tests while interacting with the conversational agent, the set of tests comprising testing a plurality of metrics including:
a performance of the conversational agent;
a linguistic variation;
a personality type of the conversation agent; and
a cognitive trait of the conversational agent; and

comparing a result of the capability from the set of tests with an expected result of the scenario based on results from a different conversational agent,
wherein the evaluation metrics are generated while interacting with the conversational system embodied in a cloud-computing environment.

11. The computer program product of claim 10 , wherein each of the at least one created scenario comprises a user input to cause an aspect of the conversational agent to be tested.

12. The computer program product of claim 10 , wherein the performing the set of tests comprises testing at least one of:
a natural language ability of the conversational agent;
pattern dialogue flow of the conversational agent;
a response to an unexpected user input by the conversational agent; and
a knowledge base of the conversational agent.

13. The computer program product of claim 10 , wherein the expected result includes at least one of:
a capability of a different conversational agent; and
a capability of a different version of a same conversational agent being tested.

14. The computer program product of claim 10 , wherein the at least one created scenario includes a speech-to-text and a text-to-speech conversion to assess the capability of the conversational agent for both of a textual input conversational agent and a speech input conversational agent.

15. The computer program product of claim 10 , further comprising storing the result from the set of tests to be used for a future comparison by the comparing.

16. The computer program product of claim 10 , further comprising generating a report of the result of the capability of the conversational agent for a human user.

17. The computer program product of claim 16 , further comprising suggesting an action for the human user to modify the conversational agent based on the report of the result of the capability of the conversational agent.

18. A conversational agent capability assessment system, said system comprising:
a processor; and
a memory, the memory storing instructions to cause the processor to perform:
obtaining data to create at least one scenario for testing a conversational agent in a conversational system having a non-deterministic output, the conversational agent being a software based on natural language classifiers;
performing a set of tests using a scenario of the at least one scenario created to assess a capability of the conversational agent based on evaluation metrics generated from the set of tests while interacting with the conversational agent, the set of tests comprising testing a plurality of metrics including:
a performance of the conversational agent;
a linguistic variation;
a personality type of the conversation agent; and
a cognitive trait of the conversational agent; and

comparing a result of the capability from the set of tests with an expected result of the scenario based on results from a different conversational agent,

wherein the evaluation metrics are generated while interacting with the conversational system embodied in a cloud-computing environment.

19. The system of claim 18 , wherein each of the at least one created scenario comprises a user input to cause an aspect of the conversational agent to be tested.

Assignees

• IBM

Inventors

• Braz Alan
• De Souza Pereira Candello Heloisa Caroline
• Pinhanez Claudio Santos
• Vasconcelos Marisa Affonso

Classifications

• G10L15/30

  Distributed recognition, e.g. in client-server systems, for mobile phones or network applications · CPC title

• G06F11/28

  by checking the correct order of processing (G06F11/08 - G06F11/26 take precedence; monitoring patterns of pulse trains H03K5/19) · CPC title

• G10L15/18

  using natural language modelling · CPC title

• G10L25/51

  for comparison or discrimination · CPC title

• G10L13/00

  Speech synthesis; Text to speech systems · CPC title