System and method for error-resilient data compression using codebooks

Patent metadata

Field	Value
Publication number	US-12079168-B2
Application number	US-202318460676-A
Country	US
Kind code	B2
Filing date	Sep 4, 2023
Priority date	Oct 30, 2017
Publication date	Sep 3, 2024
Grant date	Sep 3, 2024

Abstract

A system and method for error-resilient data reduction, utilizing a phase detector, a data requestor, a multi-phase trainer, a reconstruction engine, a deconstruction engine, and one or more reference codebooks. A multi-phase trainer may be used to train the reconstruction and deconstruction engines on various phase sourceblocks in order recover quickly from corrupted data files that cause the phase alignment of the sourceblocks to become out of phase. A phase detector may determine when the sourceblocks get out of phase and when the return to in-phase by checking if a predetermined threshold probability of correct encoding is met. Data requestor may request for retransmission only the data that was received out of phase.

First claim

What is claimed is:

1. A system for error-resilient data reduction, comprising:
a computing device comprising a memory, a processor, and a non-volatile data storage device;
a data deconstruction engine comprising a first plurality of programming instructions stored in the memory of, and operating on a processor of, the computing device, wherein the first plurality of programming instructions, when operating on the processor, cause the computing device to:
train an encoding algorithm on sourceblocks at multiple phases, wherein each phase comprises a distinct starting bit offset of a respective sourceblock;
deconstruct incoming data into a plurality of sourceblocks;
encode the sourceblocks into codewords using the encoding algorithm and a reference codebook; and
send the codewords to a data reconstruction engine; and
a data reconstruction engine comprising a second plurality of programming instructions stored in the memory of, and operating on a processor of, the computing device, wherein the second plurality of programming instructions, when operating on the processor, cause the computing device to:
decode received codewords into decoded sourceblocks using the key-value pairs stored within the reference codebook; and
for each decoded sourceblock, determine if the decoded sourceblock has exceeded a predetermined threshold probability that the decoded sourceblock was properly encoded, wherein the decoded sourceblock is in-phase if the threshold is exceeded and the decoded sourceblock is out-of-phase if the threshold is not exceeded.

2. The system of claim 1 , wherein when a decoded sourceblock is out-of-phase, the data reconstruction engine requests retransmission of codewords corresponding to out-of-phase decoded sourceblocks.

3. The system of claim 1 , wherein the threshold probability is determined using logistic regression.

4. The system of claim 1 , wherein multiple phases refer to byte-phase sourceblocks with an offset.

5. The system of claim 3 , wherein the offset is an integer value in the inclusive range of 1 to 7.

6. A method for error-resilient data reduction, comprising the steps of:
training an encoding algorithm on sourceblocks at multiple phases, wherein each phase comprises a distinct starting bit offset of a respective sourceblock;
deconstructing incoming data into a plurality of sourceblocks;
encoding the sourceblocks into codewords using an encoding algorithm and a reference codebook;
decoding the sourceblocks at a data reconstruction engine using key-value pairs stored within the reference codebook; and
for each decoded sourceblock, determining if the decoded sourceblock exceeded a predetermined threshold probability that the decoded sourceblock was properly encoded, wherein the decoded sourceblock is in-phase if the threshold is exceeded and the decoded sourceblock is out-of-phase if the threshold is not exceeded.

7. The method of claim 6 , wherein when a decoded sourceblock is out-of-phase, the data reconstruction engine requests retransmission of codewords corresponding to out-of-phase decoded sourceblocks.

8. The method of claim 6 , wherein the threshold probability is determined using logistic regression.

9. The method of claim 6 , wherein multiple phases refer to byte-phase sourceblocks with an offset.

10. The method of claim 6 , wherein the offset is an integer value in the inclusive range of 1 to 7.

Assignees

• Atombeam Technologies Inc

Inventors

• Cooper Joshua
• Riahi Aliasghar
• Haddad Mojgan
• Riahi Ryan Kourosh
• Riahi Razmin
• Yeomans Charles

Classifications

• G06F3/0608

  Saving storage space on storage systems · CPC title

• G06F3/0641

  De-duplication techniques · CPC title

• G06F3/067

  Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS] · CPC title

• G06F3/0671

  In-line storage system · CPC title

• G06F21/606

  by securing the transmission between two devices or processes · CPC title