Leveraging large language models to orchestrate microservices

Patent metadata

Field	Value
Publication number	US-2025291646-A1
Application number	US-202519078462-A
Country	US
Kind code	A1
Filing date	Mar 13, 2025
Priority date	Mar 14, 2024
Publication date	Sep 18, 2025
Grant date	—

Abstract

Systems and methods for leveraging large language models for orchestrating microservices. A cost model and a latency model can be built with a large language model (LLM) based on an instruction prompt that analyzes microservice data of a distributed computing application. The microservice data can be analyzed to learn trigger rules that facilitate placement of microservices at optimal locations within the multi-tiered computing infrastructure based on cost and latency. The microservices can be placed at the optimal locations based on the trigger rules. The LLM can be tuned using feedback from the distributed computing application after placing the microservices at the optimal locations.

First claim

What is claimed is:

1 . A computer-implemented method for leveraging large language models to orchestrate microservices, comprising:
building a cost model and a latency model by employing a large language model (LLM) based on an instruction prompt that analyzes microservice data of a distributed computing application; analyzing the microservice data with the cost model and the latency model to learn trigger rules that facilitate placement of microservices at optimal locations within a multi-tiered computing infrastructure that balances cost and latency of a workload of the distributed computing application; placing the microservices at the optimal locations based on the trigger rules; and tuning the LLM using feedback from the distributed computing application after placing the microservices at the optimal locations.

2 . The computer-implemented method of claim 1 , further comprising balancing workloads of a vehicle recognition system implemented within a multi-tiered computing infrastructure.

3 . The computer-implemented method of claim 1 , wherein building the cost model further comprises generating code snippets autonomously based on the instruction prompt.

4 . The computer-implemented method of claim 1 , wherein building the latency model further comprises generating code snippets autonomously based on the instruction prompt.

5 . The computer-implemented method of claim 1 , wherein analyzing the microservice data further comprises determining a trigger threshold for the trigger rules to facilitate placement of microservices at the optimal locations.

6 . The computer-implemented method of claim 1 , wherein analyzing the microservice data further comprises generating an optimal action based on the workload data and the trigger rules.

7 . The computer-implemented method of claim 1 , wherein tuning the LLM further comprises obtaining the feedback by measuring latency and cost from the distributed computing application with a microservice orchestration platform.

8 . A system for leveraging large language models to orchestrate microservices, comprising:
a memory device; one or more processor devices operatively coupled with the memory device to perform operations including:
building a cost model and a latency model by employing a large language model (LLM) based on an instruction prompt that analyzes microservice data of a distributed computing application;
analyzing the microservice data with the cost model and the latency model to learn trigger rules that facilitate placement of microservices at optimal locations within a multi-tiered computing infrastructure that balances cost and latency of a workload of the distributed computing application;
placing the microservices at the optimal locations based on the trigger rules; and
tuning the LLM using feedback from the distributed computing application after placing the microservices at the optimal locations.

9 . The system of claim 8 , further comprising balancing workloads of a vehicle recognition system implemented within a multi-tiered computing infrastructure.

10 . The system of claim 8 , wherein building the cost model further comprises generating code snippets autonomously based on the instruction prompt.

11 . The system of claim 8 , wherein building the latency model further comprises generating code snippets autonomously based on the instruction prompt.

12 . The system of claim 8 , wherein analyzing the microservice data further comprises determining a trigger threshold for the trigger rules to facilitate placement of microservices at the optimal locations.

13 . The system of claim 8 , wherein analyzing the microservice data further comprises generating an optimal action based on the workload data and the trigger rules.

14 . The system of claim 8 , wherein tuning the LLM further comprises obtaining the feedback by measuring latency and cost from the distributed computing application with a microservice orchestration platform.

15 . A non-transitory computer program product comprising a computer-readable storage medium including a program code, wherein the program code when executed on a computer causes the computer to perform:
building a cost model and a latency model by employing a large language model (LLM) based on an instruction prompt that analyzes microservice data of a distributed computing application; analyzing the microservice data with the cost model and the latency model to learn trigger rules that facilitate placement of microservices at optimal locations within a multi-tiered computing infrastructure that balances cost and latency of a workload of the distributed computing application; placing the microservices at the optimal locations based on the trigger rules; and tuning the LLM using feedback from the distributed computing application after placing the microservices at the optimal locations.

16 . The non-transitory computer program product of claim 15 , further comprising balancing workloads of a vehicle recognition system implemented within a multi-tiered computing infrastructure.

17 . The non-transitory computer program product of claim 15 , wherein building the cost model further comprises generating code snippets autonomously based on the instruction prompt.

18 . The non-transitory computer program product of claim 15 , wherein building the latency model further comprises generating code snippets autonomously based on the instruction prompt.

19 . The non-transitory computer program product of claim 15 , wherein analyzing the microservice data further comprises determining a trigger threshold for the trigger rules to facilitate placement of microservices at the optimal locations.

20 . The non-transitory computer program product of claim 15 , wherein analyzing the microservice data further comprises generating an optimal action based on the workload data and the trigger rules.

Assignees

• Nec Lab America Inc

Inventors

• Rao Kunal
• Coviello Giuseppe
• Benedetti Priscilla
• Devita Ciro Giuseppe
• Mellone Gennaro
• Chakradhar Srimat

Classifications

• G06F9/547

  Remote procedure calls [RPC]; Web services · CPC title

• G06F9/5027

  the resource being a machine, e.g. CPUs, Servers, Terminals · CPC title

• G06F2209/5017

  Task decomposition · CPC title

• G06F9/5072

  Grid computing · CPC title

• G06F9/505

  considering the load · CPC title