Querying in big data storage formats

Patent metadata

Field	Value
Publication number	US-11620400-B2
Application number	US-202117190786-A
Country	US
Kind code	B2
Filing date	Mar 3, 2021
Priority date	Nov 18, 2015
Publication date	Apr 4, 2023
Grant date	Apr 4, 2023

Abstract

A system may receive a request for access to a first variable. The request may include a requested action and a variable identifier for the first variable. The request may also have a syntax that is incompatible with first data storage format. The system may parse the request to capture the variable identifier from the request. The system may also look up a location of the first variable in a catalog using the variable identifier. The location of the first variable may include the first data storage format. The system may generate a query to access the first variable. The syntax of the query may be compatible with the first data storage format. The system may then submit the query to access the first data storage format. The query may be configured to complete the requested action.

First claim

What is claimed is:

1. A system, comprising:
a computing device comprising a processor and a memory; and
machine-readable instructions stored in the memory that, when executed by the processor, cause the computing device to at least:
store a first plurality of records in a first data storage format associated with a first database system accessible using a first query syntax;
store a second plurality of records in a second data storage format associated with a second database system accessible using a second query syntax;
generate a catalog of locations of respective ones of the first plurality of records and the second plurality of records, wherein individual locations in the catalog of locations represent whether a record is stored in the first database system using the first data storage format or the second database system using the second data storage format; and
associate the catalog of locations with a virtualized database structure, the virtualized database structure appearing as a unified data storage format that provides access to the first plurality of records and the second plurality of records through a third query syntax.

2. The system of claim 1 , wherein the machine-readable instructions, when executed by the processor, further cause the computing device to at least:
receive a request to access a record, wherein the request is formatted using the third query syntax provided by the virtualized database structure;
search the catalog of locations to determine whether the record is stored in the first database system using the first data storage format or the second database system using the second data storage format; and
retrieve the record from the first database system or the second database system using the respective first query syntax or the respective second query syntax.

3. The system of claim 2 , wherein the machine-readable instructions that cause the computing device to retrieve the record from the first data storage format or the second data storage format using the respective first query syntax or the respective second query syntax further cause the computing device to at least:
generate a query using the first query syntax or the second query syntax, wherein the query identifies the record specified in the request formatted using the third query syntax; and
send the query to the first database system or the second database system to retrieve the record.

4. The system of claim 1 , wherein the machine-readable instructions that cause the computing device to generate the catalog of locations of respective ones of the first plurality of records and the second plurality of records further cause the computing device to at least include metadata associated with individual ones of the first plurality of records or the second plurality of records.

5. The system of claim 4 , wherein the machine-readable instructions that cause the computing device to include metadata associated with individual ones of the first plurality of records or the second plurality of records further cause the computing device to at least copy the metadata from the first database system or the second database system to the catalog of locations.

6. The system of claim 4 , wherein the machine-readable instructions that cause the computing device to include metadata associated with individual ones of the first plurality of records or the second plurality of records further cause the computing device to at least generate the metadata for the individual ones of the first plurality of records or the second plurality of records.

7. The system of claim 1 , wherein the first plurality of records and the second plurality of records are stored in a distributed file system.

8. A computer-implemented method, comprising:
storing a first plurality of records in a first data storage format associated with a first database system accessible using a first query syntax;
storing a second plurality of records in a second data storage format associated with a second database system accessible using a second query syntax;
generating a catalog of locations of respective ones of the first plurality of records and the second plurality of records, wherein individual locations in the catalog of locations represent whether a record is stored in the first database system using the first data storage format or the second database system using the second data storage format; and
associating the catalog of locations with a virtualized database structure, the virtualized database appearing as a unified data storage format that provides access to the first plurality of records and the second plurality of records through a third query syntax.

9. The computer-implemented method of claim 8 , further comprising:
receiving a request to access a record, wherein the request is formatted using the third query syntax provided by the virtualized database structure;
searching the catalog of locations to determine whether the record is stored in the first database system using the first data storage format or the second database system using the second data storage format; and
retrieving the record from the first database system or the second database system using the respective first query syntax or the respective second query syntax.

10. The method of claim 9 , wherein retrieving the record from the first data storage format or the second data storage format using the respective first query syntax or the respective second query syntax further comprises:
generating a query using the first query syntax or the second query syntax, wherein the query identifies the record specified in the request formatted using the third query syntax; and
sending the query to the first database system or the second database system to retrieve the record.

11. The method of claim 8 , wherein generating the catalog of locations of respective ones of the first plurality of records and the second plurality of records further comprises including metadata associated with individual ones of the first plurality of records or the second plurality of records.

12. The method of claim 11 , wherein including the metadata associated with individual ones of the first plurality of records or the second plurality of records further comprises copying the metadata from the first database system or the second database system to the catalog of locations.

13. The method of claim 11 , wherein including the metadata associated with individual ones of the first plurality of records or the second plurality of records further comprises generating the metadata for the individual ones of the first plurality of records or the second plurality of records.

14. The method of claim 8 , wherein the first plurality of records and the second plurality of records are stored in a distributed file system.

15. A non-transitory, computer-readable medium comprising machine-readable instructions that, when executed by a processor of a computing device, cause the computing device to at least:
store a first plurality of records in a first data storage format associated with a first database system accessible using a first query syntax;
store a second plurality of records in a second data storage format associated with a second database system accessible using a second query syntax;
generate a catalog of locations of respective ones of the first plurality of records and the second plurality of records, wherein individual locations in the catalog of locations represent whether a record is stored in the first database system using the first data storage format or the second database system using the second data storage f

Assignees

• American Express Travel Related Services Co Inc

Inventors

• Arasan Ravi
• Argüello Carmen Patricia
• Bose Sandeep
• Meyer Matthew Kent
• Ponnambalam Marimuthu
• Ramasamy Gurusamy
• Seideman Jeremy D
• Torkzahrani Mohammad
• Varier Roopesh R

Classifications

• G06F21/6227 Primary

  where protection concerns the structure of data, e.g. records, types, queries · CPC title

• G06F16/24534

  Query rewriting; Transformation · CPC title

• G06F16/258

  Data format conversion from or to a database · CPC title

• G06F21/6218 Primary

  to a system of files or objects, e.g. local or distributed file system or database · CPC title