Pattern-aware prefetching using parallel log-structured file system

Patent metadata

Field	Value
Publication number	US-11080196-B2
Application number	US-201916716972-A
Country	US
Kind code	B2
Filing date	Dec 17, 2019
Priority date	Sep 30, 2013
Publication date	Aug 3, 2021
Grant date	Aug 3, 2021

Abstract

Techniques are provided for pattern-aware prefetching using a parallel log-structured file system. At least a portion of one or more files is accessed by detecting at least one pattern in a non-sequential access of the one or more files; and obtaining at least a portion of the one or more files based on the detected at least one pattern. The obtaining step comprises, for example, a prefetching or pre-allocation of the at least the portion of the one or more files. A prefetch cache can store the portion of the one or more obtained files. The cached portion of the one or more files can be provided from the prefetch cache to an application requesting the at least a portion of the one or more files.

First claim

What is claimed is:

1. A method for accessing at least a portion of a shared file, comprising:
performing the following steps, in response to at least one read request for at least a first portion of said shared file:
detecting at least one pattern in at least two consecutive non-sequential accesses of said shared file, wherein said at least one pattern is detected by analyzing distances between consecutive logical offsets in said at least one read request for said shared file; and
prefetching at least a second portion of said shared file from at least one storage device, wherein said second portion is predicted to be accessed based at least in part on said detected at least one pattern,
wherein said shared file is created by a plurality of processes, wherein each of said plurality of processes has a unique physical logfile for storing data written by a corresponding process and a unique physical index file for storing metadata for the corresponding process, wherein, for a given write operation of a data chunk to the shared file by a given one of said plurality of processes, (i) said data chunk is stored in said unique physical logfile corresponding to said given process, and (ii) an index entry is created for said unique physical index file corresponding to said given process, that maps said data chunk within a logical representation of said shared file to a location within said unique physical logfile, wherein a plurality of said index entries is stored in said unique physical index file corresponding to said given process in a compressed format using said detected at least one pattern and wherein said prefetching of said at least second portion of said shared file is performed using at least one of said unique physical index files, comprising said compressed format, for said plurality of processes and wherein the method is performed by at least one processing device comprising a processor coupled to a memory.

2. The method of claim 1 , wherein said prefetching step comprises a pre-allocation of said at least said second portion of said shared file.

3. The method of claim 1 , further comprising the step of storing said at least said second portion of said shared file in a prefetch cache.

4. The method of claim 3 , further comprising the step of providing said at least said second portion of said shared file from said prefetch cache to an application requesting said at least said second portion of said shared file.

5. The method of claim 4 , wherein a prefetch manager receives said detected at least one pattern from a pattern detector, predicts future requests based at least in part on said detected at least one pattern and requests a prefetch thread to fetch said at least said second portion of said shared file from a file system for storage in said prefetch cache.

6. The method of claim 4 , further comprising the step of determining whether said prefetch cache stores said second portion of said shared file requested in a read request.

7. The method of claim 1 , wherein said detecting step further comprises the steps of obtaining a plurality of read requests and detecting one or more patterns in said obtained plurality of read requests.

8. The method of claim 1 , wherein said unique physical index file indicates how said unique physical logfile for storing data written by the corresponding process was written and how said unique physical logfile for storing data written by the corresponding process is read.

9. A non-transitory processor-readable storage medium having stored therein program code of one or more software programs, wherein the program code when executed by at least one processing device causes the at least one processing device to perform the following steps in response to at least one read request for at least a first portion of a shared file:
detecting, using at least one processing device of a pattern-aware prefetching file system, at least one pattern in at least two consecutive non-sequential accesses of said shared file, wherein said at least one pattern is detected by analyzing distances between consecutive logical offsets in said at least one read request for said shared file; and
prefetching at least a second portion of said shared file from at least one storage device, wherein said second portion is predicted to be accessed based at least in part on said detected at least one pattern,
wherein said shared file is created by a plurality of processes, wherein each of said plurality of processes has a unique physical logfile for storing data written by a corresponding process and a unique physical index file for storing metadata for the corresponding process, wherein, for a given write operation of a data chunk to the shared file by a given one of said plurality of processes, (i) said data chunk is stored in said unique physical logfile corresponding to said given process, and (ii) an index entry is created for said unique physical index file corresponding to said given process, that maps said data chunk within a logical representation of said shared file to a location within said unique physical logfile, wherein a plurality of said index entries is stored in said unique physical index file corresponding to said given process in a compressed format using said detected at least one pattern and wherein said prefetching of said at least second portion of said shared file is performed using at least one of said unique physical index files, comprising said compressed format, for said plurality of processes.

10. The non-transitory processor-readable storage medium of claim 9 , wherein said prefetch manager pre-allocates said at least said second portion of said shared file.

11. The non-transitory processor-readable storage medium of claim 9 , wherein said detecting step further comprises the steps of obtaining a plurality of read requests and detecting one or more patterns in said obtained plurality of read requests.

12. The non-transitory processor-readable storage medium of claim 9 , wherein said unique physical index file indicates how said unique physical logfile for storing data written by the corresponding process was written and how said unique physical logfile for storing data written by the corresponding process is read.

13. A pattern-aware prefetching file system, comprising:
a pattern detector of a pattern-aware prefetching file system for detecting, in response to at least one read request for at least a first portion of a shared file, using at least one processing device of said pattern-aware prefetching file system, at least one pattern in at least two consecutive non-sequential accesses of said shared file, wherein said at least one pattern is detected by analyzing distances between consecutive logical offsets in said at least one read request for said shared file; and
a prefetch manager for prefetching, in response to said at least one read request for said at least first portion of said shared file, at least a second portion of said shared file from at least one storage device, wherein said second portion is predicted to be accessed based at least in part on said detected at least one pattern; and
a prefetch cache for storing at least said portion of said shared file,
wherein said shared file is created by a plurality of processes, wherein each of said plurality of processes has a unique physical logfile for storing data written by a corresponding process and a unique physical index file for storing metadata for the corresponding process, wherein, for a given write operation of a data chunk to the shared file by a given one of said plurality of processes, (i) said data chunk is stored in said unique physical logfile corresponding to sa

Assignees

• Emc Ip Holding Co Llc
• Triad Nat Security Llc

Inventors

• Bent John M
• Faibish Sorin
• Grider Gary
• Torres Aaron
• He Jun

Classifications

• G06F3/0611

  in relation to response time · CPC title

• G06F12/0862 Primary

  with prefetch · CPC title

• G06F2212/1024

  Latency reduction · CPC title

• G06F3/0643

  Management of files · CPC title

• G06F3/0659

  Command handling arrangements, e.g. command buffers, queues, command scheduling · CPC title