Migration and validation of data from tabular structures to non-relational data stores

Patent metadata

Field	Value
Publication number	US-12339869-B2
Application number	US-202217820970-A
Country	US
Kind code	B2
Filing date	Aug 19, 2022
Priority date	Aug 19, 2022
Publication date	Jun 24, 2025
Grant date	Jun 24, 2025

Abstract

In some implementations, a data migration device may querying a relational data store for a plurality of metadata items. Accordingly, for each metadata item, the data migration device may transmit, to an application programming interface (API), a request for the metadata item, and receive, from the relational data store via the API, the metadata item. Furthermore, for each metadata item, the data migration device may execute a transformation on the metadata item to generate an adapted metadata item for a non-relational data store and transmit, to the API, a request to store the adapted metadata item in the non-relational data store.

First claim

What is claimed is:

1. A system, comprising:
one or more memories; and
one or more processors, coupled to the one or more memories, configured to:
receive a command to initiate transfer from a relational data store to a non-relational data store;
transmit, to an application programming interface (API) in communication with the relational data store and the non-relational data store, a request for metadata associated with a tabular data structure in the relational data store, wherein the command includes an override header indicating that the command is associated with the relational data store;
receive, from the relational data store via the API and based on transmitting the request for the metadata, an asset identifier associated with a metadata item;
initiate, based on receiving the asset identifier, a state machine, associated with migration of the relational data store to the non-relational data store, comprising a plurality of processing threads and a plurality of states, wherein the metadata item is associated with a particular processing thread of the plurality of processing threads and a corresponding state of the plurality of states;
execute, according to the particular processing thread, to advance the state of the metadata item, a transformation on the metadata to generate, based on the metadata, adapted metadata for the non-relational data store,
wherein the transformation on the metadata is executed by converting the metadata from a first format to a second format;

transmit, to the API and according to the particular processing thread to advance the state of the metadata item, a request for storing the adapted metadata in the non-relational data store;
transmit, to the API and according to the particular processing thread to advance the state of the metadata item, a request for particular metadata associated with the non-relational data store;
receive, from the non-relational data store via the API and based on the request for the particular metadata, the particular metadata;
verify, according to the particular processing thread to advance the state of the metadata item, the particular metadata against the adapted metadata by verifying that data included in each field of the adapted metadata matches data is included in a corresponding field of the particular metadata;
determine, based on verifying the particular metadata against the adapted metadata, that there is an error with specific metadata of the particular metadata;
execute, based on determining that there is the error with the specific metadata and by converting the specific metadata from the first format to the second format, a transformation on the specific metadata to generate specific adapted metadata for the non-relational data store; and
transmit, to the API, a request for storing the specific adapted metadata in the non-relational data store.

2. The system of claim 1 , wherein the one or more processors are further configured to:
generate the override header for the request for the metadata.

3. The system of claim 1 , wherein the one or more processors are further configured to:
generate an override header for the request for storing the adapted metadata in the non-relational data store.

4. The system of claim 1 , wherein the one or more processors, to execute the transformation, are configured to:
rename one or more fields in the metadata to generate one or more adapted fields in the adapted metadata.

5. The system of claim 1 , wherein the first format is an extensible markup language format, and the second format is a JavaScript object notation format.

6. The system of claim 1 , wherein the command for the metadata includes an arbitrary input that returns all of the metadata from the tabular data structure.

7. A method, comprising:
receiving, from a user device, a command to initiate a transfer from a relational database to a non-relational database;
querying, the relational database and via an application program interface (API) in communication with the relational database and the non-relational database, the relational database for a plurality of metadata items associated with a tabular structure of the relational database;
initiating a state machine comprising a plurality of processing threads and a plurality of states, wherein each metadata item, of the plurality of metadata items, is associated with a different processing threshold of the plurality of processing threads, and a different state of the plurality of states;
for each metadata item, of the plurality of metadata items, and according to a respective processing thread, of the plurality of processing threads, and a respective state of the plurality of states:
generating an adapted metadata item based on a respective metadata item of the plurality of metadata items;
transmitting, to the API, a request for an additional metadata item from the non-relational database;
receiving, from the relational database via the API, the additional metadata item;
verifying the adapted metadata item against the additional metadata item by verifying that data included in each field of the adapted metadata item matches data included in a corresponding field of the additional metadata item;
determining, based on verifying the adapted metadata item against the additional metadata item, that there is an error with specific metadata of the metadata item;
executing, based on determining that there is the error with the specific metadata and by converting the specific metadata from a first format to a second format, a transformation on the specific metadata to generate a specific adapted metadata item for a non-relational database; and
transmitting, to the API, a request for storing the specific adapted metadata item in the non-relational database.

8. The method of claim 7 , further comprising:
updating the state machine after each specific adapted metadata item is transmitted.

9. The method of claim 7 , wherein each metadata item is associated with a tabular data structure in the relational database.

10. The method of claim 7 , wherein each adapted metadata item is associated with a document in the non-relational database.

11. The method of claim 7 , wherein executing the transformation further comprises:
renaming one or more fields in the respective metadata item to generate one or more adapted fields in the adapted metadata item.

12. The method of claim 7 , wherein the first format is an extensible markup language format, and the second format is a JavaScript object notation format.

13. A non-transitory computer-readable medium storing a set of instructions comprising:
one or more instructions that, when executed by one or more processors of a device, cause the device to:
initiate a state machine comprising a plurality of processing threads and a plurality of states;
query a relational data store for a plurality of metadata items, wherein each metadata item, of the plurality of metadata items, is associated with a particular processing threshold of the plurality of processing threads, and a particular state of the plurality of states; and
for each metadata item, of the plurality of metadata items, and according to a respective processing threshold, of the plurality of processing threads, and a respective state, of the plurality of states and corresponding to the metadata item:
generate an adapted metadata item based on a respective metadata item of the plurality of metadata items;
transmit, to an application programming interface (API), a request for an additional metadata item from a non-relational data store;
receive, from the relational data store via the API, the additional metadata i

Assignees

• Capital One Services Llc

Inventors

• Green Jeremy
• Kang Jee Won
• Zimmerman Christopher
• Murphy Brady

Classifications

• G06F16/2282

  Tablespace storage structures; Management thereof · CPC title

• G06F16/214

  Database migration support · CPC title

• G06F16/258 Primary

  Data format conversion from or to a database · CPC title