Intelligent orchestration of multimodal components

Patent metadata

Field	Value
Publication number	US-2024256948-A1
Application number	US-202318126161-A
Country	US
Kind code	A1
Filing date	Mar 24, 2023
Priority date	Jan 27, 2023
Publication date	Aug 1, 2024
Grant date	—

Abstract

In some examples, a method for orchestrating an execution plan is provided. The method includes receiving an input embedding that is generated by a machine-learning model and receiving a plurality of stored semantic embeddings, from an embedding object memory, based on the input embedding. The plurality of stored semantic embeddings each correspond to a respective historic plan. Each historic plan includes one or more executable skills. The method further includes determining a subset of semantic embeddings from the plurality of stored semantic embeddings based on a similarity to the input embedding, and generating a new plan based on the subset of semantic embeddings and the input embedding. The new plan may be different than the historic plans that correspond to the subset of semantic embeddings. The method further includes providing the new plan as an output.

First claim

What is claimed is:

1 . A method for orchestrating an execution plan, the method comprising:
receiving an input embedding, wherein the input embedding is generated by a machine-learning model; retrieving a plurality of stored semantic embeddings, from an embedding object memory, based on the input embedding, wherein the plurality of stored semantic embeddings each correspond to a respective historic plan, and wherein each historic plan comprises one or more executable skills; determining a subset of semantic embeddings from the plurality of stored semantic embeddings based on a similarity to the input embedding; generating, based on the subset of semantic embeddings and the input embedding, a new plan, wherein the new plan is different than the historic plans corresponding to the subset of semantic embeddings; and providing the new plan as an output.

2 . The method of claim 1 , wherein the new plan comprises instructions that, when executed by a computing device, cause a set of operations to be performed corresponding to one or more skills.

3 . The method of claim 1 , wherein the plurality of stored semantic embeddings each correspond to a respective historic input and the respective historic plan.

4 . The method of claim 1 , wherein the subset of semantic embeddings is further determined based on a personalization to at least one of a user or organization.

5 . The method of claim 4 , wherein the personalization comprises:
receiving metadata corresponding to the input embedding, wherein the subset of semantic embeddings are retrieved based on the similarity to the input embedding and the metadata.

6 . The method of claim 5 , wherein the metadata is associated with compliance requirements for security.

7 . The method of claim 1 , wherein the determining a subset of embeddings comprises:
determining a respective similarity between the input embedding and each embedding of the plurality of stored semantic embeddings; determining an ordered ranking of the one or more similarities or that one or more of the similarities are less than a predetermined threshold; and identifying the subset of semantic embeddings with similarities to the input embedding that are less than the predetermined threshold or based on the ordered ranking, thereby retrieving a subset of semantic embeddings from the plurality of stored semantic embeddings that is determined to be related to the input embedding.

8 . The method of claim 7 , wherein the similarities are distances.

9 . The method of claim 8 , wherein the input embedding and each embedding of the plurality of stored semantic embeddings are stored in a metric graph as nodes, wherein a respective edge is defined between the input embedding and each embedding of the plurality of stored semantic embeddings, and wherein each edge is associated with a respective distance of the distances.

10 . The method of claim 1 , further comprising, prior to receiving the input embedding:
receiving user-input; and generating the input embedding based on the user-input.

11 . A system for orchestrating an execution plan, the system comprising:
a processor; and memory storing instructions that, when executed by the processor, cause the system to perform a set of operations, the set of operations comprising:
receiving an input;
retrieving a plurality of stored semantic embeddings, from an embedding object memory, based on the input, wherein the plurality of stored semantic embeddings each correspond to respective historic plans, and wherein each historic plan comprises one or executable skills;
determining a subset of semantic embeddings from the plurality of stored semantic embeddings based on a similarity to the input;
generating, based on the subset of semantic embeddings and the input, a new plan, wherein the new plan is different than the historic plans corresponding to the subset of semantic embeddings; and
providing the new plan as an output.

12 . The system of claim 11 , wherein the input is an input embedding, and wherein the input embedding is generated by a generative multimodal machine-learning model.

13 . The system of claim 11 , wherein the set of operations further comprise:
adapting a computing device to execute the plan.

14 . The system of claim 11 , wherein the subset of semantic embeddings is further retrieved based on a personalization to at least one of a user or organization.

15 . The system of claim 14 , wherein the personalization comprises:
receiving metadata corresponding to the input, wherein the subset of semantic embeddings are retrieved based on the similarity to the input and the metadata.

16 . The system of claim 15 , wherein the metadata is associated with compliance requirements for security.

17 . The system of claim 11 , wherein the determining a subset of embeddings comprises:
determining a respective similarity between the input and each embedding of the plurality of stored semantic embeddings; determining an ordered ranking of the one or more similarities or that one or more of the similarities are less than a predetermined threshold; and identifying the subset of semantic embeddings with similarities to the input that are less than the predetermined threshold or based on the ordered ranking, thereby retrieving a subset of semantic embeddings from the plurality of stored semantic embeddings that is determined to be related to the input.

18 . The system of claim 17 , wherein the similarities are distances.

19 . The system of claim 18 , wherein the input and each embedding of the plurality of stored semantic embeddings are stored in a metric graph as nodes, wherein a respective edge is defined between the input and each embedding of the plurality of stored semantic embeddings, and wherein each edge is associated with a respective distance of the distances.

20 . A method for orchestrating an execution plan, the method comprising:
receiving an input; retrieving a plurality of stored semantic embeddings, from an embedding object memory, based on the input, wherein the plurality of stored semantic embeddings each correspond to a respective historic plan, and wherein each historic plan comprises one or more executable skills; determining a subset of semantic embeddings from the plurality of stored semantic embeddings based on a similarity to the input; filtering the subset of semantic embeddings based on a personalization associated with at least one of a user or organization, wherein the personalization is based on metadata associated with compliance requirements for security; generating, based on the filtered subset of semantic embeddings and the input, a new plan, wherein the new plan is different than the historic plans corresponding to the subset of semantic embeddings; and providing the new plan as an output.

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Betthauser Leo Moreno
• Blum William
• Wicker Andrew W
• Douglas Eric Paul
• Greenwald Lloyd Geoffrey
• Becker Nicholas

Classifications

• G06N20/00 Primary

  Machine learning · CPC title