Automatically generating conversational services from a computing application

Patent metadata

Field	Value
Publication number	US-10705892-B2
Application number	US-201816002915-A
Country	US
Kind code	B2
Filing date	Jun 7, 2018
Priority date	Jun 7, 2018
Publication date	Jul 7, 2020
Grant date	Jul 7, 2020

Abstract

The automatic generation of one or more task-oriented conversational bots is disclosed. Illustratively, systems and methods are provided that allow for tracing the interactions of one or more computing applications inclusive of the interaction with one or more programmatic elements of the one or more computing applications, interaction with the graphical user interface(s) of the one or more computing applications, and/or the operation of the one or more computing environments on which the one or more computing applications are executing to collect various state data. The state data can be illustratively graphed to show the overall execution paths of one or more functions/operations of the one or more computing applications for use in generating one or more instructions representative of a desired task-oriented conversational bot that can be operatively executed through one or more application program interfaces of the one or more computing applications.

First claim

What is claimed is:

1. A system comprising:
at least one processor; and
at least one memory having computer-readable instructions stored thereupon that, when executed by the at least one processor, cause the at least one processor to:
receive first input data comprising data representative of one or more user interactions with one or more programmatic elements, data representative of the one or more programmatic elements, and data representative of a state of an application in which the one or more of the programmatic elements are being executed;
process the first input data to generate intermediate processed input data by:
partitioning the first input data according to one or more selected application states application;
identifying one or more dependencies among the one or more selected application states; and
extracting one or more service application program interfaces (APIs) from the application using an identified one or more states and one or more dependencies; and
generating a user interface using the intermediate processed input data by:
generating one or more questions for the identified one or more dependencies;
generating samples of user answers for the generated one or more questions; and
generating sample of user utterances for triggering the extracted one or more service APIs to be executed by the application.

2. The system of claim 1 , wherein the one or more user interactions comprise one or more traces of a behavior of the application.

3. The system of claim 1 , wherein the data representative of the one or more programmatic elements comprises data representative of a user interface of the application.

4. The system of claim 1 , wherein the computer-readable instructions further cause the at least one processor to partition the first input data according to one or more selected application states comprising one or more states of a user interface of an application executable on a computing environment and one or more states representative of one or more internal processing states of the application executable on the computing environment.

5. The system of claim 1 , wherein data representative of the state of the application in which the programmatic element is being executed comprises dynamic application state, static application state, and the state of an underlying operating system for a computer environment on which the application is executed.

6. The system of claim 1 , wherein the computer-readable instructions further cause the at least one processor to order the one or more selected application states as one or more elements of a state graph, the state graph comprising data representative of the one or more dependencies among the one or more selected application states.

7. The system of claim 6 , wherein the computer-readable instructions further cause the at least one processor to extract the one or more service APIs as one or more paths on the state graph, the one or more paths comprising data representative of the one or more selected states of the application.

8. A computer-implemented method, comprising:
receiving first input data by a computing environment comprising data representative of one or more user interactions with one or more programmatic elements, data representative of the one or more programmatic elements, and data representative of a state of an application in which the one or more programmatic elements are being executed;
processing the first input data to generate intermediate processed input data by:
partitioning the first input data according to one or more selected application states;
ordering the one or more selected application states as one or more elements of a state graph, wherein the state graph comprises data representative of one or more dependencies among the one or more selected application states; and
extracting one or more service application program interfaces (APIs) as one or more paths of the state graph, the one or more paths comprising data representative of one or more parameters for the extracted one or more service APIs; and

generating a natural user interface using the intermediate processed input data by:
generating one or more questions for an identified parameter;
generating samples of one or more user answer to the generated one or more questions; and
generating samples of one or more user utterances for the extracted one or more service APIs to be executed by the application.

9. The computer-implemented method of claim 8 , further comprising performing a trace of the application operative on the computing environment, the trace generating data representative of processed data by one or more functions of the application.

10. The computer-implemented method of claim 8 , wherein partitioning the first input data according to one or more selected application states comprises partitioning the first input data according to one or more states of a user interface of the application and one or more internal processing states of the application.

11. The computer-implemented method of claim 8 , wherein one or more paths comprise data representative of one or more states of the application.

12. The computer-implemented method of claim 11 , further comprising generating a skill and providing the skill to one or more user channels to execute, the one or more user channels comprising a cooperating computing application.

13. The computer-implemented method of claim 8 , further comprising identifying one or more service APIs from the received input data or user request.

14. The computer-implemented method of claim 8 , further comprising training the computing environment to generate the one or more questions for the identified parameter, to generate the samples of one or more user answers to the generated one or more questions, and to generate the samples of one or more user utterances for the identified action or service API.

15. A non-transitory computer-readable storage medium having computer-executable instructions stored thereupon which, when executed by one or more processors of a computing device, cause the one or more processors of the computing device to:
receive first input data comprising data representative of one or more user interactions with a programmatic element, data representative of the programmatic elements, and data representative of a state of an application in which the programmatic element is being executed;
process the first input data to generate intermediate processed input data by:
partitioning the first input data according to one or more selected application states;
identifying one or more dependencies among the one or more selected application states; and
extracting one or more service application program interfaces (APIs) from the application using the identified one or more dependencies; and

generating a natural user interface using the intermediate processed input data by:
generating one or more questions for the identified one or more dependencies;
generating samples of user answers for the generated one or more questions; and
generating samples of one or more utterances for the extracted one or more service APIs to be executed by the application.

16. The non-transitory computer-readable storage medium of claim 15 , wherein the instructions further cause the one or more processors of the computing device to:
trace generating data representative of processed data by one or more functions of the application.

17. The non-transitory computer-readable storage medium of claim 16 , wherein the instructions further cause the one or more processors of the

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Riva Oriana
• Kace Jason Alan
• Burger Douglas Christopher
• Li Jiajun

Classifications

• G06N3/042

  Knowledge-based neural networks; Logical representations of neural networks · CPC title

• G06F9/543 Primary

  User-generated data transfer, e.g. clipboards, dynamic data exchange [DDE], object linking and embedding [OLE] · CPC title

• G06N5/022 Primary

  Knowledge engineering; Knowledge acquisition · CPC title

• G06F9/451

  Execution arrangements for user interfaces · CPC title

• G06N5/041

  Abduction · CPC title