Generating a unifiled knowledge graph from multimodal data sources using collaborative multi-agent large language models

Patent metadata

Field	Value
Publication number	US-2026093701-A1
Application number	US-202519334072-A
Country	US
Kind code	A1
Filing date	Sep 19, 2025
Priority date	Sep 30, 2024
Publication date	Apr 2, 2026
Grant date	—

Abstract

Existing multi-agent system are employed collaboratively, where each agent specialized in extracting entities, relations, or events from documents which is a more generalized collaborative framework without specific customizations. The disclosed method presents a collaborative multi-agent knowledge graph Retrieval-Augmented Generation (RAG) framework using a plurality of collaborative multi-agent LLMs for extracting information from a plurality of multimodal documents and constructing the unified knowledge graph to facilitate accurate and efficient query-based retrieval. A query generator agent generates a plurality of queries that aim to uncover all possible information present in the plurality of multimodal documents. A domain model is generated by a domain model generator agent based on the plurality of queries. The domain model populator agent populates the domain model with data extracted from the plurality of multimodal documents. A unified knowledge graph is constructed from ta plurality of populated domain models and organizes the extracted information for accurate retrieval.

First claim

What is claimed is:

1 . A processor implemented method, the method comprising:
receiving, via one or more hardware processors, a plurality of multimodal documents, and a plurality of domain historical questions pertaining to the plurality of multimodal documents; processing, via the one or more hardware processors, the plurality of multimodal documents to identify text and a plurality of images, using a python library; chunking, via the one or more hardware processors, the text and the plurality of images into a plurality of chunks comprising a plurality of text chunks, and a plurality of image chunks; and generating, via the one or more hardware processors, a unified knowledge graph from the plurality of chunks, using a plurality of collaborative multi-agent large language models (LLMs), wherein the plurality of collaborative multi-agent LLMs comprises a query generator agent, a domain model generator agent, a domain model populator agent, and a knowledge graph curator agent, wherein generating the unified knowledge graph comprises:
(i) generating a plurality of queries associated with each chunk of the plurality of chunks, through the query generator agent comprises:
(a) building a query agent, by defining a plurality of query generator behavioral parameters, and a plurality of query generator functional specification parameters through an input JavaScript Object Notation (JSON) model, wherein the plurality of query generator behavioural parameters comprises a query generator role, an agent knowledge enriched using the plurality of domain historical questions, a query generator interface, and wherein the plurality of query generator functional specification parameters comprises a plurality of query generator actions, a plurality of tasks comprising a plurality of query generator tasks and a plurality of query generator evaluation tasks, a feedback from the plurality of collaborative multi-agent LLMs, and a plurality of query generator tools;
(b) formulating a query prompt template using a plurality of instructions, the plurality of chunks, an action, and the query agent;
(c) generating a query prompt using the query prompt template; and
(d) feeding the query prompt to the query generator agent, to generate the plurality of queries associated with each chunk of the plurality of chunks;

(ii) generating a domain model associated with each chunk of the plurality of chunks based on the generated plurality of queries, through the domain model generator agent;
(iii) populating the generated domain model using each chunk of the plurality of chunks, to generate a populated domain model of a plurality of populated domain models for each chunk of the plurality of chunks, through the domain model populator agent; and
(iv) generating the unified knowledge graph associated with each chunk of the plurality of chunks, using the plurality populated domain models, through the knowledge graph curator agent.

2 . The processor implemented method of claim 1 , wherein the generated unified knowledge graph is queried through a knowledge graph query agent by:
(a) receiving one or more queries pertaining to the plurality of multimodal documents from one or more users; (b) building a unified knowledge graph query agent, by defining a plurality of knowledge graph query behavioral parameters, and a plurality of knowledge graph query functional specification parameters, wherein the plurality of knowledge graph query behavioral parameters comprises a domain model generator role, a domain knowledge, and a domain model generator interface, and wherein the plurality of knowledge graph query functional specification parameters comprises a plurality of domain model generator actions, a plurality of domain model generator tasks comprising a plurality of knowledge graph query tasks and a plurality of knowledge graph query evaluation tasks, the feedback from the plurality of collaborative multi-agent LLMs, and a plurality of domain model generator tools; (c) formulating a knowledge graph query template using the plurality of instructions, a schema model, the one or more queries, and the unified knowledge graph query agent; (d) generating a knowledge graph query prompt from the knowledge graph query prompt template; (e) feeding the knowledge graph query prompt to the knowledge graph query agent, to generate a cypher query for a knowledge graph engine; (f) generating a response comprising a graph data retrieved using a pattern matching from the unified knowledge graph, by the knowledge graph engine; and (g) processing the response from the knowledge graph engine along with one or more queries and a prompt to convert the graph data to a human interpretable natural language, using a Large Language Model (LLM).

3 . The processor implemented method of claim 1 , wherein the domain model associated with each chunk of the plurality of chunks based on the generated plurality of queries, through the domain model generator agent is generated by:
(a) building a domain model agent, by defining a plurality of domain model generator behavioral parameters, and a plurality of domain model generator functional specification parameters, wherein the plurality domain model generator behavioral parameters comprises the domain model generator role, the domain knowledge, the domain model generator interface, and wherein the plurality of domain model generator functional specification parameters comprises the plurality of domain model generator actions, the plurality of domain model generator tasks comprising a plurality of domain model generator query tasks and a plurality of domain model generator evaluation tasks, the feedback from the plurality of collaborative multi-agent LLMs, and the plurality of domain model generator tools; (b) formulating a domain model generator prompt template using the plurality of instructions, the plurality of queries associated with each chunk of the plurality of chunks, and the action, and the domain model agent; (c) generating a domain model generator prompt from the domain model generator prompt template; and (d) feeding the domain model generator prompt to the domain model generator agent, to generate the domain model associated with each chunk of the plurality of chunks based on the generated plurality of queries.

4 . The processor implemented method of claim 1 , wherein the generated domain model is populated, using each chunk of the plurality of chunks, to generate a populated domain model of a plurality of populated domain models for each chunk of the plurality of chunks, through the domain model populator agent comprises:
(a) building a domain model populating agent, by defining a plurality of domain model populator behavioral parameters, and a plurality of domain model populator functional specification parameters, wherein the plurality of domain model populator behavioral parameters comprise a domain model populator role, the domain knowledge, and a domain model populator interface, and wherein the domain model populator functional specification parameters comprise a plurality of domain model populator actions, a plurality of domain model populator tasks comprising a plurality of domain model populator query tasks and a plurality of domain model populator evaluation tasks, the feedback from the plurality of collaborative multi-agent LLMs, and a plurality of domain model populator tools; (b) formulating a domain model populator prompt template using the plurality of instructions, the plurality of chunks, the domain model, the action, and the domain model populating agent; (c) generating a domain model populator prompt from the domain model populator prompt template; and (d) feeding the domain model populator prompt to the domain model populator agent, to generate the plurality of populated domain models associated with each chu

Assignees

• Tata Consultancy Services Ltd

Inventors

• Krishna Ankur
• Ardham Surya
• Malhotra Chetan Premkumar

Classifications

• G06N3/042

  Knowledge-based neural networks; Logical representations of neural networks · CPC title

• G06F16/367

  Ontology · CPC title

• G06F16/243

  Natural language query formulation · CPC title

• G06F16/2455 Primary

  Query execution · CPC title

• G06F16/33295 Primary

  in dialogue systems · CPC title