Querying Data Records Stored On A Distributed File System

Patent metadata

Field	Value
Publication number	US-2018060341-A1
Application number	US-201615254467-A
Country	US
Kind code	A1
Filing date	Sep 1, 2016
Priority date	Sep 1, 2016
Publication date	Mar 1, 2018
Grant date	—

Abstract

Systems and methods for query large database records are disclosed. An example method includes: obtaining a first search query including a first keyword; accessing a relational database that stores a mapping between one or more keywords and a data record location associated with a distributed file system (DFS). The data record location identifies a location on the DFS at which a data record matching the one or more keywords is stored. The method also includes, determining, using a relational database, a first data record location based on the first keyword; identifying a first data record based on the first data record location; and providing the first data record as a matching record responsive to the first search query.

First claim

What is claimed is:

1 . A method, comprising:
obtaining a first search query including a first keyword; accessing a relational database that stores a mapping between one or more keywords and a data record location associated with a distributed file system (DFS), wherein the data record location identifies a location on the DFS at which a data record matching the one or more keywords is stored; determining, using the relational database, a first data record location based on the first keyword; identifying a first data record based on the first data record location; and providing the first data record as a matching record responsive to the first search query.

2 . The method of claim 1 , wherein the mapping is an inverted index mapping from the one or more keywords to the data record location.

3 . The method of claim 1 , further comprising: retrieving, as part of a batch data processing, the first data record from the DFS.

4 . The method of claim 1 , wherein the search query includes a second keyword different from the first keyword; and further comprising: determining, using the relational database, the first data record location based on the second keyword.

6 . The method of claim 1 , further comprising:
obtaining a second search query including a second keyword; determining, using the relational database, a second data record location based on the second keyword; identifying a second data record based on the second data record location; executing a batch data retrieval job to retrieve the first data record and the second data record; and providing the second data record as a matching record responsive to the second search query.

6 . The method of claim 1 , further comprising: acknowledging that the first search query has a first matching record store on the DFS.

7 . The method of claim 6 , wherein the acknowledging occurs as part of a stream data processing job.

8 . The method of claim 1 , wherein the DFS system includes a Hadoop database and the relational database is a SQL database.

9 . The method of claim 1 , wherein the one or more keywords include a plurality of keywords.

10 . A system, comprising:
a non-transitory memory; and one or more hardware processors coupled to the non-transitory memory and configured to execute instructions to perform operations comprising: receiving a first search query including a first keyword; receiving a second search query including a second keyword; accessing a relational database that stores a mapping between one or more keywords and a data record location associated with a distributed file system (DFS), wherein the data record location identifies a location on the DFS at which a data record matching the one or more keywords is stored; determining, using the relational database, a first data record location based on the first keyword and a second data record location based on the second keyword; identifying a first data record based on the first data record location and a second data record based on the second data record location; and performing a batch data processing job to retrieve the first data record and the second data record from the DFS.

11 . The system of claim 10 , wherein the operations further comprise:
retrieving the first data record from a first data node associated with the DFS; and retrieving the second data record from a second data node associated with the DFS.

12 . The system of claim 10 , wherein the operations further comprising: responsive to determining the first data record location and the second data record location, acknowledging that matching records exist for the first search query and the second search query.

13 . The system of claim 10 , wherein receiving the first search query and receiving the second search query are part of a stream data processing job.

14 . The system of claim 10 , wherein the first data record and the second data records are greater than a predefined file size.

16 . A non-transitory machine-readable medium having stored thereon machine-readable instructions executable to cause a machine to perform operations comprising:
obtaining a first search query including a first keyword; obtaining a second search query including a second keyword; accessing a relational database that stores a mapping between one or more keywords and a data record location associated with a distributed file system (DFS), wherein the data record location identifies a location on the DFS at which a data record matching the one or more keywords is stored; determining, using the relational database, a first data record location based on the first keyword and a second data record location based on the second keyword; identifying a first data record based on the first data record location and a second data record based on the second data record location; and performing a batch data processing job to retrieve the first data record and the second data record from the DFS.

16 . The non-transitory machine-readable medium of claim 16 , wherein performing the batch data processing job comprises:
requesting a name node to retrieve the first data record based on the first data record location and to retrieve the second data record based on the second data record location.

17 . The non-transitory machine-readable medium of claim 16 , wherein the operations further comprise:
retrieving the first data record and the second data record from a same data node associated with the DFS.

18 . The non-transitory machine-readable medium of claim 16 , wherein the first query includes a request to modify the first data record based on the first keyword.

19 . The non-transitory machine-readable medium of claim 16 , wherein the one or more keywords include a plurality of keywords.

20 . The non-transitory machine-readable medium of claim 16 , wherein the DFS system includes a Hadoop database and the relational database is a SQL database.

Assignees

• Paypal Inc

Inventors

• Wu Haifeng
• Zhang Pengshan
• Shen Wei

Classifications

• G06F16/134

  Distributed indices · CPC title

• G06F16/182

  Distributed file systems · CPC title

• G06F16/148 Primary

  File search processing · CPC title

• G06F17/30194

  Physics · mapped topic

• G06F17/30094

  Physics · mapped topic