System and method for information retrieval using dynamic sharding

Patent metadata

Field	Value
Publication number	US-2025317490-A1
Application number	US-202418629748-A
Country	US
Kind code	A1
Filing date	Apr 8, 2024
Priority date	Apr 8, 2024
Publication date	Oct 9, 2025
Grant date	—

Abstract

A distributed data storage includes data stored in different data shards on nodes of the distributed data storage. A memory stores a dynamic heatmap indicative of a number of data requests for data stored on each of the nodes. A processor receives a request for retrieving the data stored in the distributed data storage. The request includes data ranges and index criteria of the data and location of the nodes in the distributed data storage. In response, the processor determines the nodes of the distributed data storage that include the data using the dynamic heatmap, retrieves the data as determined from the dynamic heatmap; and routes the data to a user associated with the request. When a data retrieval performance is below a desired threshold value, the processor performs dynamic data sharding to reallocate the data shards across the nodes of the distributed data storage.

First claim

1 . A system, comprising:
a distributed data storage including data stored in different data shards on nodes of the distributed data storage; a memory configured to store a dynamic heatmap indicative of a number of data requests for the data stored on each of the nodes, frequency of the data requests, processing time of the data requests and response times of the nodes for the data requests; and a processor communicatively coupled to the memory, the processor configured to:
receive a request for retrieving at least a portion of the data stored in the distributed data storage, wherein the request includes data ranges and index criteria of the data and location of the nodes in the distributed data storage;
determine the nodes of the distributed data storage that include the data indicated by the data ranges, index criteria and locations included in the request using the dynamic heatmap;
retrieve, based on a state of a network including the distributed data storage as determined using roaming filters and based on the dynamic heatmap, the data from the nodes of the distributed data storage; and
route the data to a user associated with the request, through the network including the distributed data storage, wherein, in response to determining that a data retrieval performance of retrieving data from the nodes is below a desired threshold, the processor is further configured to perform dynamic data sharding to reallocate the different data shards across the nodes of the distributed data storage.

2 . The system of claim 1 , wherein the processor is further configured to collect a plurality of requests from different sources and consolidate similar requests from the plurality of requests into a single dataset.

3 . The system of claim 1 , wherein the processor is further configured to perform data sharding by reallocating data shards of the data that are more frequently accessed than other data shards of the data (1) to nodes of the distributed data storage that are geographically closer to the user, (2) to nodes of the distributed data storage that are less frequently accessed than other nodes of the distributed data storage, or (3) to nodes of the distributed data storage that are accessible by the user in a lesser time than other nodes of the distributed data storage.

4 . The system of claim 1 , wherein the roaming filters are configured to run multiple parallel processes to retrieve network data simultaneously to determine a state of the network.

5 . The system of claim 1 , wherein the processor is further configured to analyze the request using a request mediator to obtain the data ranges and the index criteria from the request, wherein the processor is further configured to generate multiple threads, using the request mediator, to fetch data for each of the determined data range and index criteria.

6 . The system of claim 5 , wherein the processor is further configured to receive a plurality of data requests and assign a dedicated request mediator to each data request of the plurality of data requests.

7 . The system of claim 1 , wherein the processor is further configured to route the data using contextual information associated with the data, wherein the contextual information comprises one or more of a type of the data, priority of the data, user preferences, network conditions, and current workload of the network.

8 . A method, comprising:
storing, in a memory, a dynamic heatmap indicative of a number of data requests for data stored on each node of a distributed data storage, frequency of the data requests, processing time of the data requests and response times of the nodes for the data requests, wherein the data is stored in different data shards on the nodes of the distributed data storage; receiving a request for retrieving at least a portion of the data stored in the distributed data storage, wherein the request includes data ranges and index criteria of the data and location of the nodes in the distributed data storage; determining the nodes of the distributed data storage that include the data indicated by the data ranges, index criteria and locations included in the request using the dynamic heatmap; retrieving, based on a state of a network including the distributed data storage as determined using roaming filters and based on the dynamic heatmap, the data from the nodes of the distributed data storage; routing the data to a user associated with the request, through a network including the distributed data storage; and in response to determining that a data retrieval performance of retrieving data from the nodes is below a desired threshold, performing data sharding to reallocate the different data shards across the nodes of the distributed data storage.

9 . The method of claim 8 , further comprising collecting a plurality of requests from different sources and consolidating similar requests from the plurality of requests into a single dataset.

10 . The method of claim 8 , further comprising performing data sharding by reallocating data shards of the data that are more frequently accessed than other data shards of the data (1) to nodes of the distributed data storage that are geographically closer to the user, (2) to nodes of the distributed data storage that are less frequently accessed than other node of the distributed data storage, or (3) to nodes of the distributed data storage that are accessible by the user in a lesser time than other nodes of the distributed data storage.

11 . The method of claim 8 , wherein the roaming filters are configured to run multiple parallel processes to retrieve network data simultaneously to determine a state of the network.

12 . The method of claim 8 , further comprising:
analyzing the request using a request mediator to obtain the data ranges and the index criteria from the request; and generating multiple threads, using the request mediator, to fetch data for each of the determined data range and index criteria.

13 . The method of claim 12 , further comprising:
receiving a plurality of data requests; and assigning a dedicated request mediator to each data request of the plurality of data requests.

14 . The method of claim 8 , further comprising routing the data using contextual information associated with the data, wherein the contextual information comprises one or more of a type of the data, priority of the data, user preferences, network conditions, and current workload of the network.

15 . A non-transitory computer-readable medium storing instructions that when executed by a processor cause the processor to:
store, in a memory, a dynamic heatmap indicative of a number of data requests for data stored on each node of a distributed data storage, frequency of the data requests, processing time of the data requests and response times of the nodes for the data requests, wherein the data is stored in different data shards on the nodes of the distributed data storage; receive a request for retrieving at least a portion of the data stored in the distributed data storage, wherein the request includes data ranges and index criteria of the data and location of the nodes in the distributed data storage; determine the nodes of the distributed data storage that include the data indicated by the data ranges, index criteria and locations included in the request using the dynamic heatmap; retrieve, using roaming filters, the data from the nodes as determined from the dynamic heatmap; route the data to a user associated with the request, through a network including the distributed data storage; and in response to determining that a data retrieval per

Assignees

• Bank Of America

Inventors

• Adivi Suryanarayana
• Taneja Pushkar
• Singh Shailendra
• Ryali Venkata

Classifications

• H04L67/1097 Primary

  for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS] · CPC title