Intelligent data ingestion chunk size optimization

Patent metadata

Field	Value
Publication number	US-12438823-B2
Application number	US-202318503632-A
Country	US
Kind code	B2
Filing date	Nov 7, 2023
Priority date	Nov 7, 2023
Publication date	Oct 7, 2025
Grant date	Oct 7, 2025

Abstract

An embodiment establishes a transmission performance database that comprises performance data related to transmission of data across a plurality of data channels of a network. The embodiment defines an initial data chunk size for a first data object transmitted across a first data channel of the plurality data channels of the network. The embodiment transmits the first data object across the first data channel according to the initial data chunk size. The embodiment collects performance data for the first data channel. The embodiment evaluates transmission performance of the first data channel based at least in part on the performance data of the first data channel. The embodiment generates a recommendation to adjust the initial data chunk size of the first data object based on the transmission performance of the first data channel. The embodiment adjusts the initial data chunk size based at least in part on the recommendation.

First claim

What is claimed is:

1. A computer-implemented method comprising:
establishing a transmission performance database, wherein the transmission performance database comprises historical performance data related to transmission of data across a plurality of data channels of a network;
establishing a chunk size adjustment schedule based on the historical performance data stored on the transmission performance database;
defining an initial data chunk size for a first data object transmitted across a first data channel of the plurality data channels of the network, wherein the initial data chunk size is defined according to the chunk size adjustment schedule;
transmitting the first data object across the first data channel according to the initial data chunk size;
collecting performance data for the first data channel data and storing the performance data for the first data channel on the performance database;
evaluating transmission performance of the first data channel based at least in part on the performance data of the first data channel;
generating a recommendation to adjust the initial data chunk size of the first data object based on the transmission performance of the first data channel; and
adjusting, upon acceptance of the recommendation, the initial data chunk size based at least in part on the recommendation.

2. The computer-implemented method of claim 1 , wherein adjusting the initial data chunk size comprises increasing the initial data chunk size.

3. The computer-implemented method of claim 1 , wherein the method further comprises:
defining an initial data chunk size for a second data object transmitted across a second data channel of the plurality data channels of the network;
transmitting the second data object across the second data channel according to the initial data chunk size for the second data object transmitted across the second data channel;
collecting performance data for the second data channel data and storing the performance data for the second data channel on the performance database;
evaluating transmission performance of the second data channel based at least in part on the performance data of the second data channel;
generating a recommendation to adjust the initial data chunk size of the second data object based on the transmission performance of the second data channel; and
upon acceptance of the recommendation, adjusting the initial data chunk size of the second data object based at least in part on the recommendation.

4. The computer-implemented method of claim 3 , wherein adjusting the initial chunk size for the first data object comprises incrementing the initial data chunk size by a different amount than incrementing the initial data chunk size for the second data object.

5. The computer-implemented method of claim 1 , further comprising training a machine learning algorithm to increment data chunk size based at least in part on historical performance data stored on the transmission performance database.

6. The computer-implemented method of claim 1 , further comprising dynamically adjusting data chunk size based at least in part on historical performance data stored on the transmission performance database.

7. The computer-implemented method of claim 1 , further comprising adjusting data chunk size according to a predetermined schedule.

8. The computer-implemented method of claim 1 , wherein evaluating transmission performance of the first data channel comprises comparing at least one performance metric against a performance metric threshold, and upon a determination that the at least one performance metric meets the performance metric threshold, generating a recommendation to increase a current data chunk size.

9. A computer program product comprising one or more computer readable storage media, and program instructions collectively stored on the one or more computer readable storage media, the program instructions executable by a processor to cause the processor to perform operations comprising:
establishing a transmission performance database, wherein the transmission performance database comprises historical performance data related to transmission of data across a plurality of data channels of a network;
establishing a chunk size adjustment schedule based on the historical performance data stored on the transmission performance database;
defining an initial data chunk size for a first data object transmitted across a first data channel of the plurality data channels of the network, wherein the initial data chunk size is defined according to the chunk size adjustment schedule;
transmitting the first data object across the first data channel according to the initial data chunk size;
collecting performance data for the first data channel data and storing the performance data for the first data channel on the performance database;
evaluating transmission performance of the first data channel based at least in part on the performance data of the first data channel;
generating a recommendation to adjust the initial data chunk size of the first data object based on the transmission performance of the first data channel; and
adjusting, upon acceptance of the recommendation, the initial data chunk size based at least in part on the recommendation.

10. The computer program product of claim 9 , wherein the stored program instructions are stored in a computer readable storage device in a data processing system, and wherein the stored program instructions are transferred over the network from a remote data processing system.

11. The computer program product of claim 9 , wherein the stored program instructions are stored in a computer readable storage device in a server data processing system, and wherein the stored program instructions are downloaded in response to a request over the network to a remote data processing system for use in a computer readable storage device associated with the remote data processing system, further comprising:
program instructions to meter use of the program instructions associated with the request; and
program instructions to generate an invoice based on the metered use.

12. The computer program product of claim 9 , wherein adjusting the initial data chunk size comprises increasing the initial data chunk size.

13. The computer program product of claim 9 , further comprising:
defining an initial data chunk size for a second data object transmitted across a second data channel of the plurality data channels of the network;
transmitting the second data object across the second data channel according to the initial data chunk size for the second data object transmitted across the second data channel;
collecting performance data for the second data channel data and storing the performance data for the second data channel on the performance database;
evaluating transmission performance of the second data channel based at least in part on the performance data of the second data channel;
generating a recommendation to adjust the initial data chunk size of the second data object based on the transmission performance of the second data channel; and
upon acceptance of the recommendation, adjusting the initial data chunk size of the second data object based at least in part on the recommendation.

14. The computer program product of claim 13 , wherein adjusting the initial chunk size for the first data object comprises incrementing the initial data chunk size by a different amount than incrementing the initial data chunk size for the second data object.

15. The computer program product claim 9 , further comprising training a machine lear

Assignees

• IBM

Inventors

• Thomas Binoy
• Kairali Sudheesh S
• Uhlig Hans W
• Rakshit Sarbajit K

Classifications

• H04L43/16

  Threshold monitoring · CPC title

• H04L43/0888

  Throughput · CPC title

• H04L41/16

  using machine learning or artificial intelligence · CPC title

• H04L47/365 Primary

  Dynamic adaptation of the packet size · CPC title