LLM integrations for content analysis in software applications

Patent metadata

Field	Value
Publication number	US-12524627-B2
Application number	US-202318313857-A
Country	US
Kind code	B2
Filing date	May 8, 2023
Priority date	May 8, 2023
Publication date	Jan 13, 2026
Grant date	Jan 13, 2026

Abstract

Technology is disclosed herein for the integration of large language model (LLM) services for content analysis in productivity applications. In an implementation, an application identifies content associated with the application executing on the computing apparatus. The application identifies a context relating to the content, wherein the context comprises contextual information by which to evaluate the content. The application generates a prompt for an LLM service which includes the content and the context. The prompt requests an evaluation of the content to recommend supplemental content and submits the prompt to the LLM service. The application receives a response to the prompt from the LLM service which includes a suggestion for supplemental content based on the evaluation and displays, in a user interface of the application, a recommendation based on the suggestion from the LLM service.

First claim

What is claimed is:

1 . A computing apparatus comprising:
one or more computer-readable storage media; one or more processors operatively coupled with the one or more computer-readable storage media; and program instructions stored on the one or more computer-readable storage media that, when executed by the one or more processors, direct the computing apparatus to at least:
identify content of a document associated with an application executing on the computing apparatus;
receive, via a user interface of the application, user input comprising a context relating to the content, wherein the context comprises information by which to evaluate the content;
submit, to a large language model (LLM) service, a first prompt including the content of the document, wherein the first prompt tasks the LLM service with generating an inferred purpose of the content and a confidence score associated with the inferred purpose;
receive a first response comprising the inferred purpose and the confidence score;
determine that the confidence score exceeds a threshold value;
in response to determining that the confidence score exceeds the threshold value, select a prompt template based on the inferred purpose;
generate a second prompt for the LLM service based on the prompt template, wherein the second prompt includes the content of the document and the context, and wherein the second prompt requests an evaluation of the content based on the context and a suggestion based on the evaluation;
submit the second prompt to the LLM service;
receive a second response to the second prompt from the LLM service, wherein the response comprises the suggestion based on the evaluation;
enable display, in the user interface of the application, of a recommendation comprising the suggestion from the LLM service; and
receive, via the user interface, user input indicative of a request to implement the recommendation in the document.

2 . The computing apparatus of claim 1 , wherein the application comprises a productivity application.

3 . The computing apparatus of claim 2 , wherein the program instructions further direct the computing apparatus to:
display the inferred purpose generated by the LLM service in the user interface; and receive, via the user interface, a user input comprising a confirmation of the inferred purpose.

4 . The computing apparatus of claim 3 , wherein the second prompt further requests the evaluation of the content based on comparing the content with model content generated by the LLM service based on the context.

5 . The computing apparatus of claim 4 , wherein to enable display of the recommendation in the user interface of the productivity application, the program instructions direct the computing apparatus to display the recommendation in a task pane of the user interface, and wherein the program instructions further direct the computing apparatus to receive, in the task pane, the user input indicative of the request to implement the recommendation in the document.

6 . The computing apparatus of claim 5 , wherein the context includes document metadata.

7 . The computing apparatus of claim 6 , wherein the content comprises information relating to a user profile of a user associated with the document, and wherein the program instructions further direct the application to identify, from the document, a source of information relating to the user profile and to collect the information relating to the user profile from the source, wherein the source is different from the application.

8 . The computing apparatus of claim 1 , wherein the content comprises information relating to a user profile of a user and the context comprises subject matter of a tutorial.

9 . The computing apparatus of claim 1 , wherein the prompt template comprises an instruction to the LLM service to generate the second response in a specified format, and wherein to enable the display of the recommendation in the user interface, the program instructions direct the computing apparatus to parse the second response from the LLM service to extract the suggestion from the second response according to the specified format for display in the user interface.

10 . A method of operating an application service, the method comprising:
identifying content of a document associated with the application service executing on a computing device; receiving, via a user interface of the application service, user input comprising a context relating to the content, wherein the context comprises contextual information by which to evaluate the content; submitting, to a large language model (LLM) service, a first prompt including the content of the document, wherein the first prompt tasks the LLM service with generating an inferred purpose of the content and a confidence score associated with the inferred purpose; receiving a first response comprising the inferred purpose and the confidence score; determining that the confidence score exceeds a threshold value; in response to determining that the confidence score exceeds the threshold value, select a prompt template based on the inferred purpose; generating a second prompt for the LLM service based on the prompt template, wherein the second prompt includes the content of the document and the context, and wherein the second prompt requests an evaluation of the content based on the context and a suggestion based on the evaluation; submitting the second prompt to the LLM service; receiving a second response to the second prompt from the LLM service, wherein the response comprises the suggestion based on the evaluation; enabling display, in the user interface of the application service, of a recommendation based on the suggestion from the LLM service; and receiving, via the user interface, user input indicative of a request to implement the recommendation in the document.

11 . The method of claim 10 , wherein the application service comprises a productivity application.

12 . The method of claim 11 , further comprising
displaying the inferred purpose generated by the LLM service in the user interface; and receiving, via the user interface, a user input comprising a confirmation of the inferred purpose.

13 . The method of claim 12 , wherein the second prompt requests the evaluation of the content based on comparing the content with model content generated by the LLM service based on the context.

14 . The method of claim 13 , wherein enabling display of the recommendation in the user interface of the application service further comprises displaying the recommendation in a task pane of the user interface, and wherein the method further comprises receiving, in the task pane, the user input indicative of the request to implement the recommendation in the document.

15 . The method of claim 14 , wherein the context includes document metadata.

16 . The method of claim 14 , wherein the context comprises information relating to a user profile of a user associated with the content, and wherein the method further comprises: identifying from the document a source of information relating to the user profile; and collecting the information relating to the user profile from the source, wherein the source is different from the application service.

17 . The method of claim 10 , wherein the content comprises information relating to a user profile of a user and the context comprises subject matter of a tutorial.

18 . A method of operating an application service, the method comprising:
identifying content associated with the application se

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Bhupati Dhirendra Kumar
• Sharma Mrinal Kumar

Classifications

• G06F40/20

  Natural language analysis (semantic analysis of natural language G06F40/30) · CPC title

• G06F9/451

  Execution arrangements for user interfaces · CPC title

• G06F16/383

  using metadata automatically derived from the content · CPC title

• G06F40/40 Primary

  Processing or translation of natural language (natural language analysis G06F40/20; semantic analysis G06F40/30) · CPC title

• G06F16/335 Primary

  Filtering based on additional data, e.g. user or group profiles (filtering in web context G06F16/9535, G06F16/9536) · CPC title