File list generation method, system, and program, and file list generation device

Patent metadata

Field	Value
Publication number	US-8965941-B2
Application number	US-201313753208-A
Country	US
Kind code	B2
Filing date	Jan 29, 2013
Priority date	Feb 15, 2012
Publication date	Feb 24, 2015
Grant date	Feb 24, 2015

Abstract

A file list generation device creates, at high speed, a list of added, changed, or deleted file data by acquiring the directory URLs of search target files from a first storage, counting the number of directories in each layer through analysis of the directory URLs, and, based on the result of the counting, determines a dividing policy to divide a scan target directory scanning operation, which is to be performed with a distributed server cluster. Directories of the search target files are divided in accordance with the determined dividing policy, and a snapshot of a current directory tree is acquired.

First claim

What is claimed is:

1. A file list generation method, comprising the steps of:
storing index information created about search target files into a first storage area;
determining a dividing policy to divide a scan target directory scanning operation to be performed with a distributed server cluster, by acquiring directory URLs of the search target files from the index information stored in the first storage area, and counting the number of directories in each layer through analysis of the directory URLs, the scan target directory scanning operation being divided based on the result of the counting;
dividing directories of the search target files in accordance with the determined dividing policy;
acquiring a snapshot of a current directory tree of scan target files, and storing the snapshot into a second storage area;
scanning, in a distributed manner, the search target files in respective directories in the snapshot of the current directory tree and a snapshot of a directory tree of search target files acquired prior to the determination of the divided policy and stored in a third storage area, the scanning being performed with the distributed server cluster on the basis of division units determined in the determining the dividing policy; and
comparing the results of the distributed scans, and creating and outputting a list of files that are added, deleted, or changed after the creation of the index information.

2. A file list generation method, comprising the steps of:
storing index information created about search target files into a first storage area;
copying a current directory tree of the search target files, and storing the copied current directory tree as a backup directory into a second storage area;
determining a dividing policy to divide a scan target directory scanning operation to be performed with a distributed server cluster, by acquiring directory URLs of the search target files from the index information stored in the first storage area, and counting the number of directories in each layer through analysis of the directory URLs, the scan target directory scanning operation being divided based on the result of the counting;
dividing the backup directory of the search target files in accordance with the determined dividing policy;
scanning, in a distributed manner, the search target files in respective directories in the backup directory of the current directory tree and a backup directory tree of search target files acquired prior to the determination of the divided policy and stored in a third storage area, the scanning being performed with the distributed server cluster on the basis of division units determined in the determining the dividing policy; and
comparing the results of the distributed scans, and creating and outputting a list of files that are added, deleted, or changed after the creation of the index information.

3. The file list generation method according to claim 1 , wherein the step of determining the dividing policy includes the steps of
determining a dividing policy of setting the number of division units as one to perform a search target directory scanning operation when the number of directories counted from the highest layer of directories of search target files is smaller than a predetermined number,
performing a first scan, with each one division unit being layers from a directory in the highest layer to a predetermined layer when the number of directories counted from the highest layer exceeds the predetermined number, and
determining a dividing policy to scan lower layers of scan target directories that are found through the first scan, with each one division unit being a depth of one layer.

4. The file list generation method according to claim 2 , wherein the step of determining the dividing policy includes the steps of
determining a dividing policy of setting the number of division units as one to perform a search target directory scanning operation when the number of directories counted from the highest layer of directories of search target files is smaller than a predetermined number,
performing a first scan, with each one division unit being layers from a directory in the highest layer to a predetermined layer when the number of directories counted from the highest layer exceeds the predetermined number, and
determining a dividing policy to scan lower layers of scan target directories that are found through the first scan, with each one division unit being a depth of one layer.

5. A file list generation system comprising:
a processor; and
a first storage area, connected to said processor, that stores index information created about search target files and instructions that, when executed by the processor, instruct the processor to perform acts including:
acquiring directory URLs of the search target files from the index information stored in the first storage area, counting the number of directories in each layer through analysis of the directory URLs, and, based on the counting, determining a dividing policy to divide a scan target directory scanning operation to be performed with a distributed server cluster;
dividing directories of the search target files in accordance with the dividing policy, acquires a snapshot of a current directory tree of scan target files, and storing the snapshot into a second storage area;
scanning, in a distributed manner, the search target files in respective directories in the snapshot of the current directory tree and a snapshot of a directory tree of search target files acquired prior to the determination of the divided policy and stored in a third storage area, the scanning being performed with the distributed server cluster on the basis of division units determined by the dividing policy; and
comparing the results of the distributed scans performed by the scanning, and creating and outputting a list of files that are added, deleted, or changed after the creation of the index information.

6. A file list generation system comprising:
a processor;
a first storage area, connected to said processor, that stores index information created about search target files; and
a second storage area, connected to said processor, that copies a current directory tree of the search target files, and stores the copied current directory tree as a backup directory,
wherein the first storage area stores instructions that, when executed by the processor, instruct the processor to perform acts including:
acquiring directory URLs of the search target files from the index information stored in the first storage area, counting the number of directories in each layer through analysis of the directory URLs, and, based on the counting, determining a dividing policy to divide a scan target directory scanning operation to be performed with a distributed server cluster;
dividing the backup directory of the search target files in accordance with the determined dividing policy;
scanning, in a distributed manner, the search target files in respective directories in the backup directory of the current directory tree stored in the second storage area and a backup directory tree of search target files acquired prior to the determination of the divided policy and stored in a third storage area, the scanning being performed with the distributed server cluster on the basis of division units determined in the dividing policy; and
comparing the results of the distributed scans, and creates and outputs a list of files that are added, deleted, or changed after the creation of the index information.

7. The file list generation system according to claim 5 , wherein the determining of the dividing policy includes:
determining a dividing policy of setting the number of division units as one to perform a search tar

Assignees

• Hitachi Solutions Ltd

Inventors

• Nishida Shimpei
• Imai Takashi

Classifications

• G06F17/30091 Primary

  Physics · mapped topic

• G06F17/30144

  Physics · mapped topic

• G06F17/3007

  Physics · mapped topic

• G06F16/13 Primary

  File access structures, e.g. distributed indices (arrangements of input from, or output to, record carriers G06F3/06) · CPC title

• G06F16/11 Primary

  File system administration, e.g. details of archiving or snapshots (error detection or correction of the data by redundancy in operations G06F11/14) · CPC title