Data processing performance enhancement in a distributed file system

Patent metadata

Field	Value
Publication number	US-9405692-B2
Application number	US-201213426466-A
Country	US
Kind code	B2
Filing date	Mar 21, 2012
Priority date	Mar 21, 2012
Publication date	Aug 2, 2016
Grant date	Aug 2, 2016

Abstract

Systems and methods of data processing performance enhancement are disclosed. One embodiment includes, invoking operating system calls to optimize cache management by an I/O component; wherein, the operating system calls are invoked to perform one or more of; proactive triggering of readaheads for sequential read requests of a disk; purging data out of buffer cache after writing to the disk or performing sequential reads from the desk; and/or eliminating a delay between when a write is performed and when written data from the write is flushed to the disk from the buffer cache.

First claim

What is claimed is:

1. A method for enhancing performance for data processing in a distributed file system, the method comprising:
instantiating an input/output (I/O) manager on a machine among a plurality of machines that implement the distributed file system; and
utilizing the I/O manager to perform cache management optimization including:
(a) determining that the machine employs a heuristic for triggering readaheads for sequential read requests;
overriding the heuristic so as to deterministically trigger the readaheads for all sequential read requests;
(b) determining that the machine is configured to automatically cache data into a buffer on the machine after the data is accessed;
detecting that a specific size of data has been accessed;
instructing the machine to invalidate the cached data in the buffer; and
(c) determining that the machine is configured to include a time delay before committing data from the buffer to a disk on the machine;
overriding the time delay so that the machine commits the data from the buffer to the disk without the time delay.

2. The method of claim 1 , wherein, the cache management is optimized for reads and writes in Hbase or MapReduce.

3. The method of claim 1 , wherein, the cache management is optimized for large sequential reads and writes.

4. The method of claim 1 , wherein an amount of the readaheads for sequential read requests is dynamically adjusted based on the sequential read requests.

5. The method of claim 1 , wherein the method further comprising:
instantiating a connection manager on the machine, wherein the connection manager is configured to instruct the machine perform steps including:
upon receiving an access request from a client, establishing a connection with a node associated with the access request in response to the access request;
determining that the access request is associated with data that is smaller than a select size;
detecting whether operations requested by the access request have been completed;
after the operations requested by the access request have been completed, holding the connection with the node open for a select time period,
receiving, by the connection manager from the client, a subsequent access request that is associated with the node associated with the access request before the select time period expires; and
performing operations requested by the subsequent access request through the held open connection.

6. The method of claim 1 , wherein, data size of the data that is invalidated from the buffer is configurable or dynamically adjustable.

7. The method of claim 1 , wherein, data size of the data that is invalidated from the buffer is between 4-8 MB or 8-16 MB.

8. The method of claim 1 , wherein the cache management optimization are performed through operating system calls are native to an operating system running on the machine.

9. The method of claim 1 , further comprising decreasing checksum overhead in a read path of the distributed file system by modifying a polynomial that is used in generating the checksum.

10. The method of claim 9 , wherein, the checksum is performed in hardware by a processor supporting CRC32.

11. The method of claim 1 , wherein, the operating system is Linux or Unix-based operating systems.

12. The method of claim 1 , wherein, the operating system conforms to POSIX.1-2001.

13. The method of claim 1 , wherein, the distributed file system is the Hadoop distributed file system.

14. A system for distributed computing, the system comprising:
a set of machines forming a distributed file system cluster, a given machine in the set of machines having:
a processor;
a disk;
memory having stored there on instructions which when executed by the processor, causes the given machine to perform:
(a) determining that the machine employs a heuristic for triggering readaheads for sequential read requests;
overriding the heuristic so as to deterministically trigger the readaheads for all sequential read requests;
(b) determining that the machine is configured to automatically cache data into a buffer on the machine after the data is accessed;
detecting that a specific size of data has been accessed;
instructing the machine to invalidate the cached data in the buffer; and
(c) determining that the machine is configured to include a time delay before committing data from the buffer to a disk on the machine;
overriding the time delay so that the machine commits the data from the buffer to the disk without the time delay.

15. The system of claim 14 , wherein the given machine is further caused to perform:
instantiating a connection manager on the machine, wherein the connection manager is configured to instruct the machine perform steps including:
upon receiving an access request from a client, establishing a connection with node associated with the access request in response to the access request;
determining that the access request is associated with data that is smaller than a select size;
detecting whether operations requested by the access request have been completed;
after the operations requested by the access request have been completed, holding the connection with the node open for a select time period,
receiving, by the connection manager from the client, a subsequent access request that is associated with the node associated with the access request before the select time period expires; and
performing operations requested by the subsequent access request through the held open connection.

16. The system of claim 14 , wherein the connection is held open for 0.5-1 seconds, 1-2, seconds, or 2-5 seconds.

17. The system of claim 14 , wherein an amount of the readaheads for sequential read requests is dynamically adjusted based on the sequential read requests.

18. The system of claim 14 , wherein, the distributed file system is the Hadoop distributed file system (HDFS).

19. The system of claim 14 , wherein, the cache management is optimized for reads and writes in Hbase or MapReduce.

20. The system of claim 14 , wherein the given system is further cased to perform:
decreasing checksum overhead in a read path of the distributed file system by modifying a polynomial that is used in generating the checksum.

21. The system of claim 20 , wherein, the checksum uses CRC32C algorithm.

Assignees

• Lipcon Todd
• Cloudera Inc

Inventors

• Lipcon Todd

Classifications

• G06F16/182 Primary

  Distributed file systems · CPC title

• G06F2212/214

  Solid state disk · CPC title

• G06F12/0871

  Allocation or management of cache space · CPC title

• G06F12/0804

  with main memory updating (G06F12/0806 takes precedence) · CPC title

• G06F13/20

  for access to input/output bus · CPC title