Verification of backup data across a data pipeline

Patent metadata

Field	Value
Publication number	US-12430317-B2
Application number	US-202418426634-A
Country	US
Kind code	B2
Filing date	Jan 30, 2024
Priority date	Jan 30, 2024
Publication date	Sep 30, 2025
Grant date	Sep 30, 2025

Abstract

Systems, devices, and techniques are disclosed for verification of backup data across a data pipeline. Records from a first storage may be received at a first end of a data pipeline. The records may be hashed to generate first hashes. A first hash tree may be generated from the first hashes. The records may be received at a second end of the data pipeline. Bits of bitmaps that correspond to the records may be set. The records may be hashed to generate second hashes. The records may be stored in a second storage. A second hash tree may be generated form the second hashes. Using the bitmaps, whether all of the records or any duplicate records were received may be determined. The first hash tree and the second hash tree may be compared to determine if any of the records stored in the second storage are corrupt.

First claim

The invention claimed is:

1. A computer-implemented method comprising:
receiving at a first end of a data pipeline, from a first storage, records;
hashing the records to generate first hashes;
generating a first hash tree from the first hashes;
receiving, at a second end of the data pipeline, the records;
generating bitmaps for unique batches of the records;
setting bits of the bitmaps that correspond to the records received at the second end of the data pipeline as the records are received at the second end of the data pipeline;
hashing the records as the records are received at the second end of the data pipeline to generate second hashes;
storing the records in a second storage as the records are received at the second end of the data pipeline;
generating a second hash tree from the second hashes;
determining, using the bitmaps, if all of the records were received at the second end of the data pipeline and if any duplicates of any of the records were received at the second end of the data pipeline; and
comparing the first hash tree and the second hash tree to determine if any of the records stored in the second storage are corrupt.

2. The computer-implemented method of claim 1 , further comprising preventing access to the records in the second storage until the comparison of the first hash tree and the second hash tree determines that none of the records in the second storage are corrupt.

3. The computer-implemented method of claim 1 , further comprising sending a record to the second end of the data pipeline when it is determined using the bitmaps that the record was not received at the second of the data pipeline.

4. The computer-implemented method of claim 1 , further comprising assigning the records batch identifiers and ordinal identifiers after the records are received at the first end of the data pipeline and before the records are received at the second end of the data pipeline.

5. The computer-implemented method of claim 4 , further comprising removing the batch identifiers and the ordinal identifiers from the records before storing the records in the second storage.

6. The computer-implemented method of claim 1 , wherein generating the second hash tree further comprises using the batch identifiers and the ordinal identifiers from the records.

7. The computer-implemented method of claim 1 , wherein the records further comprise object identifiers, and further comprising generating additional records for the unique batches of the records, the additional records comprising batch identifiers, starting object identifiers, and ending object identifiers.

8. A computer-implemented system comprising:
a first storage and a second storage; and
one or more processors that receive at a first end of a data pipeline, from the first storage, records,
hash the records to generate first hashes,
generate a first hash tree from the first hashes,
receive, at a second end of the data pipeline, the records;
generate bitmaps for unique batches of the records;
set bits of the bitmaps that correspond to the records received at the second end of the data pipeline as the records are received at the second end of the data pipeline,
hash the records as the records are received at the second end of the data pipeline to generate second hashes;
store the records in the second storage as the records are received at the second end of the data pipeline;
generate a second hash tree from the second hashes,
determine, using the bitmaps, if all of the records were received at the second end of the data pipeline and if any duplicates of any of the records were received at the second end of the data pipeline, and
compare the first hash tree and the second hash tree to determine if any of the records stored in the second storage are corrupt.

9. The computer-implemented system of claim 8 , wherein the one or more processors further prevent access to the records in the second storage until the comparison of the first hash tree and the second hash tree determines that none of the records in the second storage are corrupt.

10. The computer-implemented system of claim 8 , wherein the one or more processors further send a record to the second end of the data pipeline when it is determined using the bitmaps that the record was not received at the second of the data pipeline.

11. The computer-implemented system of claim 8 , wherein the one or more processors assign the records batch identifiers and ordinal identifiers after the records received at the first end of the data pipeline and before the records are received at the second end of the data pipeline.

12. The computer-implemented system of claim 8 , wherein the one or more processors further remove the batch identifiers and the ordinal identifiers from the records before storing the records in the second storage.

13. The computer-implemented system of claim 8 , wherein the one or more processors generate the second hash tree by using the batch identifiers and the ordinal identifiers from the records.

14. The computer-implemented system of claim 8 , wherein the records further comprise object identifiers, and wherein the one or more processors further generate additional records for the unique batches of the records, the additional records comprising batch identifiers, starting object identifiers, and ending object identifiers.

15. A system comprising: one or more computers and one or more non-transitory storage devices storing instructions which are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:
receiving at a first end of a data pipeline, from a first storage, records;
hashing the records to generate first hashes;
generating a first hash tree from the first hashes;
receiving, at a second end of the data pipeline, the records;
generating bitmaps for unique batches of the records;
setting bits of the bitmaps that correspond to the records received at the second end of the data pipeline as the records are received at the second end of the data pipeline;
hashing the records as the records are received at the second end of the data pipeline to generate second hashes;
storing the records in a second storage as the records are received at the second end of the data pipeline;
generating a second hash tree from the second hashes;
determining, using the bitmaps, if all of the records were received at the second end of the data pipeline and if any duplicates of any of the records were received at the second end of the data pipeline; and
comparing the first hash tree and the second hash tree to determine if any of the records stored in the second storage are corrupt.

16. The system of claim 15 , wherein the one or more computers and one or more non-transitory storage devices further store instructions which are operable, when executed by the one or more computers, to cause the one or more computers to further perform operations comprising:
preventing access to the records in the second storage until the comparison of the first hash tree and the second hash tree determines that none of the records in the second storage are corrupt.

17. The system of claim 15 , wherein the one or more computers and one or more non-transitory storage devices further store instructions which are operable, when executed by the one or more computers, to cause the one or more computers to further perform operations comprising:
sending a record to the second end of the data pipeline when it is determined using the bitmaps that the record was not received at the second of the da

Assignees

• Salesforce Inc

Inventors

• Zhang Lionel
• Taylor Zachary

Classifications

• G06F11/1458

  Management of the backup or restore process · CPC title

• G06F2201/84

  Using snapshots, i.e. a logical point-in-time copy of the data · CPC title

• G06F11/1448

  Management of the data involved in backup or backup restore · CPC title

• G06F16/2365 Primary

  Ensuring data consistency and integrity · CPC title