Digital assistant extension automatic ranking and selection

Patent metadata

Field	Value
Publication number	US-11379489-B2
Application number	US-201916572000-A
Country	US
Kind code	B2
Filing date	Sep 16, 2019
Priority date	Oct 10, 2016
Publication date	Jul 5, 2022
Grant date	Jul 5, 2022

Abstract

Representative embodiments disclose mechanisms to automatically rank and select extensions triggered in a digital assistant. A sample set of extensions are executed against a set of curated queries in order to extract a set of features and/or statistics. The system trains a machine learning model based on the features and/or statistics to rank and select extensions based on their response to a query. New extension incorporated into the system are executed against a second set of curated queries to obtain a set of extracted features and/or statistics which are saved for use at runtime. At runtime, a query phrase received by the system triggers one or more tasks from extensions. Extracted features for the triggered extensions are combined with stored features/statistics and at least a subset of the results presented to the trained ranking and selection model. The model ranks and selects appropriate tasks which are presented to the user.

First claim

What is claimed is:

1. A method for selecting a bot of a digital assistant from amongst several bots of the digital assistant, the method comprising:
receiving a query from a user that is directed to the digital assistant;
identifying the several bots based upon the query;
providing, for the several bots, respective features assigned to the several bots
to a machine learning model, wherein the features are based upon responses generated by the several bots to queries previously received by the several bots, wherein the machine learning model has been trained based on profiles of a plurality of bots, and further wherein the machine learning model generates scores for the several bots based upon the features assigned to the several bots;
selecting the bot from the several bots based upon the scores generated by the machine learning model; and
presenting an output of the bot to the user based upon the bot being selected, wherein the output is presented by the bot based upon the query.

2. The method of claim 1 , wherein the machine learning model has been trained based upon a profile of the bot, and further wherein the profile comprises features of the bot that are based upon responses of the bot to the queries previously received by the bot.

3. The method of claim 2 , wherein the profile of the bot is based upon a dialog comprising a series of interactions with the bot.

4. The method of claim 2 , wherein the profile of the bot further comprises security checks made prior to deployment of the bot.

5. The method of claim 1 , wherein the features assigned to the several bots comprise at least one of:
features extracted from results of a language understanding model;
features extracted from how well a conversation progress under a hypothesis that a specific response is accepted and presented to the user; or
features extracted from statistics corresponding to the bot.

6. The method of claim 1 , wherein a score for the bot from amongst the scores is generated by the machine learning model based further upon at least one of:
a correlation metric between a language understanding model of the bot and a built-in language understanding model;
a list of tasks that the bot handles; or
at least one metric describing at least one query that causes the bot to trigger.

7. The method of claim 1 , wherein the queries previously received by the bot comprise a set of curated queries, the method further comprising:
prior to receiving the query:
retrieving the set of curated queries;
executing the set of curated queries against the bot;

gathering features for the bot based upon responses of the bot to the set of curated queries; and
building the profile based on the features for the bot.

8. The method of claim 7 , further comprising:
prior to receiving the query, extracting from the bot a set of features comprising tasks handled by the bot, wherein the machine learning model generates a score for the bot from amongst the scores based upon the set of features.

9. The method of claim 7 , further comprising:
identifying a set of positive trigger examples or a set of negative trigger examples or both based on the responses of the bot to the previously received queries, wherein the features for the bot are based upon the set of positive trigger examples, the set of negative trigger examples, or both.

10. The method of claim 1 , wherein the machine learning model generates a score for the bot based upon a set of features for the bot identified at run-time.

11. A computing system comprising:
a processor; and
memory storing instructions that, when executed by the processor, cause the processor to perform acts comprising:
receiving a query from a user that is directed to a digital assistant;
identifying, based upon the query, several bots associated with the digital assistant;
providing features assigned to a bot in the several bots to a machine learning model, wherein the features are based upon responses generated by the bot to queries previously received by the bot, wherein the machine learning model has been trained based on profiles of a plurality of bots, and further wherein the machine learning model computes a score for the bot based upon the features assigned to the bot;
selecting the bot from the several bots based upon the score for the bot computed by the machine learning model; and
presenting an output of the bot to the user based upon the bot being selected, wherein the output is presented by the bot as a response to the query.

12. The computing system of claim 11 , wherein the profiles of the plurality of bots include a profile of the bot, and further wherein the profile of the bot includes the features assigned to the bot.

13. The computing system of claim 12 , wherein the profile of the bot is based upon a dialog comprising a series of interactions with the bot.

14. The computing system of claim 12 , wherein the profile of the bot further comprises security checks made prior to deployment of the bot.

15. The computing system of claim 11 , wherein the features assigned to the bot comprise at least one of:
features extracted from results of a language understanding model;
features extracted from how well a conversation progress under a hypothesis that a specific response is accepted and presented to the user; or
features extracted from statistics corresponding to the bot.

16. The computing system of claim 11 , wherein a score for the bot is computed based further upon at least one of:
a correlation metric between a language understanding model of the bot and a built-in language understanding model;
a list of tasks that the bot handles; or
at least one metric describing at least one query that causes the bot to trigger.

17. The computing system of claim 11 , wherein the queries previously received by the bot comprise a set of curated queries, the acts further comprising:
prior to receiving the query:
retrieving the set of curated queries;
executing the set of curated queries against the bot;
gathering the features assigned to the bot based upon responses of the bot to the set of curated queries; and
building a profile for the bot based on the features assigned to the bot.

18. The computing system of claim 17 , the acts further comprising:
prior to receiving the query, extracting from the bot a set of features comprising tasks handled by the bot, wherein the machine learning model computes the score based upon the set of features.

19. The computing system of claim 17 , the acts further comprising:
identifying a set of positive trigger examples or a set of negative trigger examples or both based on the responses of the bot to the previously received queries, wherein the features assigned to the bot are based upon the set of positive trigger examples, the set of negative trigger examples, or both.

20. A machine readable medium comprising instructions that, when executed by a processor, cause the processor to perform acts comprising:
receiving a query from a user that is directed to a digital assistant;
identifying, based upon the query, several bots associated with the digital assistant;
providing features assigned to a bot in the several bots to a machine learning model, wherein the features are based upon responses generated by the bot to queries previously received by the bot, wherein the machine learning model has been trained based on profiles of a plurality of bots, and further wherein the machine learning model computes a score for the bot based upon the features as

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Robichaud Jean-Philippe

Classifications

• G06F16/9535

  Search customisation based on user profiles and personalisation · CPC title

• G06N20/00

  Machine learning · CPC title

• H04L51/02 Primary

  using automatic reactions or user delegation, e.g. automatic replies or chatbot-generated messages · CPC title

• G06F16/24578 Primary

  using ranking · CPC title

• H04L51/214

  using selective forwarding · CPC title