Systems and methods for compressing data for distributed ledgers

Patent metadata

Field	Value
Publication number	US-12182128-B2
Application number	US-202318151969-A
Country	US
Kind code	B2
Filing date	Jan 9, 2023
Priority date	Jan 9, 2023
Publication date	Dec 31, 2024
Grant date	Dec 31, 2024

Abstract

System and methods for compressing data for use in a distributed ledger are provided. In one form, the system includes: a network interface; a user interface device; and a control circuit coupled to the network interface and the user interface device. In this form, the control circuit executes a data compression module configured to: receive, via the network interface, a data set; transform and store the data set into a file format of rows and columns in which data of the data set are stored in a columnar fashion; apply a first compression step to compress each row and write each compressed row into the file format; apply a second compression step to further compress data after the compressed rows have been written into the file format; and encode the compressed data and transmit the encoded data to a first node of a distributed ledger.

First claim

What is claimed is:

1. A system for compressing data for use in a distributed ledger, the system comprising:
a network interface;
and
a control circuit coupled to the network interface, the control circuit executing a data compression module configured to:
receive, via the network interface, a data set;
transform and store the data set into a file format of rows and columns in which data of the data set are stored in a columnar fashion;
apply a first compression step to compress each row and write each compressed row into the file format;
apply a second compression step to further compress data after the compressed rows have been written into the file format; and
encode the compressed data and transmit the encoded data to a first node of a distributed ledger;
wherein the first compression step and the second compression step apply different compression algorithms, the first compression step applying row-level compression and the second compression step applying file-level compression.

2. The system of claim 1 , wherein the data set comprises a temporal series of measurements.

3. The system of claim 1 , wherein the control circuit is configured to transform and store the data of the data set in a parquet file format.

4. The system of claim 3 , wherein the control circuit is configured to use a Snappy compression algorithm in the first compression step to compress each row and write each compressed row into the parquet file format.

5. The system of claim 4 , wherein the control circuit is configured to use a gzip algorithm in the second compression step to further compress data in the parquet file format.

6. The system of claim 5 , wherein the control circuit is configured to encode the further compressed data by creating a base64 string encoding of the data in the parquet file format.

7. The system of claim 1 , wherein the control circuit is configured to share the encoded data, via the first node of the distributed ledger and the network interface, with a second node of the distributed ledger, the first and second nodes corresponding to a first entity and a second entity, respectively.

8. The system of claim 7 , wherein the control circuit is configured to:
decode the encoded data into the file format in which the data are stored in a columnar fashion;
apply a first decompression step to unzip the decoded data; and
apply a second decompression step to transform the unzipped data to correspond to the received data set.

9. The system of claim 7 , wherein the first entity is a purchaser of products and the second entity is a carrier of the products.

10. The system of claim 9 , wherein the control circuit is configured to:
compare data in the data set to a predetermined threshold corresponding to a characteristic of products transported by the carrier on behalf of the purchaser;
determine that at least a portion of the data exceeds the predetermined threshold; and
transmit a claim communication from the first entity to the second entity regarding the products transported by the carrier.

11. The system of claim 1 , further comprising a user interface device, the control circuit coupled to the user interface device.

12. A method for compressing data for use in a distributed ledger, the method comprising:
receiving, by a control circuit executing a data compression module, a data set via a network interface;
transforming and storing, by the control circuit, the data set into a file format of rows and columns in which data of the data set are stored in a columnar fashion;
applying, by the control circuit, a first compression step to compress each row and write each compressed row into the file format;
applying, by the control circuit, a second compression step to further compress data after the compressed rows have been written into the file format; and
encoding, by the control circuit, the compressed data and transmitting the encoded data to a first node of a distributed ledger;
wherein the first compression step and the second compression step apply different compression algorithms, the first compression step applying row-level compression and the second compression step applying file-level compression.

13. The method of claim 12 , wherein the data set comprises a temporal series of measurements.

14. The method of claim 12 , further comprising, by the control circuit:
transform and store the data of the data set in a parquet file format.

15. The method of claim 14 , further comprising, by the control circuit:
using a Snappy compression algorithm in the first compression step to compress each row and write each compressed row into the parquet file format.

16. The method of claim 15 , further comprising, by the control circuit:
using a gzip algorithm in the second compression step to further compress data in the parquet file format.

17. The method of claim 16 , further comprising, by the control circuit:
encoding the further compressed data by creating a base64 string encoding of the data in the parquet file format.

18. The method of claim 12 , further comprising, by the control circuit:
sharing the encoded data, via the first node of the distributed ledger and the network interface, with a second node of the distributed ledger, the first and second nodes corresponding to a first entity and a second entity, respectively.

19. The method of claim 18 , further comprising, by the control circuit:
decoding the encoded data into the file format in which the data are stored in a columnar fashion;
applying a first decompression step to unzip the decoded data; and
applying a second decompression step to transform the unzipped data to correspond to the received data set.

20. The method of claim 18 , wherein the first entity is a purchaser of products and the second entity is a carrier of the products;
the method further comprising, by the control circuit:
comparing data in the data set to a predetermined threshold corresponding to a characteristic of products transported by the carrier on behalf of the purchaser;
determining that at least a portion of the data exceeds the predetermined threshold; and
transmitting a claim communication from the first entity to the second entity regarding the products transported by the carrier.

Assignees

• Walmart Apollo Llc

Inventors

• Hu Lingyun
• Banik Anand
• Parab Pushparaj
• Gunawardana Haritha
• Folck Travis B

Classifications

• G06F16/221 Primary

  Column-oriented storage; Management thereof · CPC title

• G06F16/258

  Data format conversion from or to a database · CPC title

• G06F16/24561 Primary

  Intermediate data storage techniques for performance improvement · CPC title