Method for classifying an unmanaged dataset

Patent metadata

Field	Value
Publication number	US-10055430-B2
Application number	US-201514882983-A
Country	US
Kind code	B2
Filing date	Oct 14, 2015
Priority date	Oct 14, 2015
Publication date	Aug 21, 2018
Grant date	Aug 21, 2018

Abstract

A computer implemented method for classifying at least one source dataset of a computer system. The method may include providing a plurality of associated reference tables organized and associated in accordance with a reference storage model in the computer system. The method may also include calculating, by a data classifier application of the computer system, a first similarity score between the source dataset and a first reference table of the reference tables based on common attributes in the source dataset and a join of the first reference table with at least one further reference table of the reference tables having a relationship with the first reference table. The method may further include classifying, by the data classifier application, the source dataset by determining using at least the calculated first similarity score whether the source dataset is organized as the first reference table in accordance to the reference storage model.

First claim

What is claimed is:

1. A computer implemented method for classifying at least one source dataset of a computer system, the method comprising:
providing a plurality of associated reference tables organized and associated in accordance with a reference storage model in the computer system;
calculating, by a data classifier application of the computer system, a first similarity score between the source dataset and a first reference table of the reference tables based on common attributes in the source dataset and a join of the first reference table with at least one further reference table of the reference tables having a relationship with the first reference table, wherein the first similarity score is calculated with the following formula,

Score
=

∑

dist
=
0

n

⁢

⁢

S
dist

1
+
dist

,

where
⁢

⁢

S
dist

=

∑

d
=
0

ds

⁢

⁢

2
*

W
d

N
DS

+

N
DT

,

ds is the number of common attributes, Wd=1/card(D) where card(D) is the cardinality of an attribute D in the reference tables, NDS is the number of attributes in the source dataset, NDT is the number of attributes in the reference tables, n is the number of the at least one further reference table plus the first reference table, and dist is the distance in terms of number of foreign key relationships between the first reference table and the at least one further reference table; and
classifying, by the data classifier application, the source dataset by determining using at least the calculated first similarity score whether the source dataset is organized as the first reference table in accordance to the reference storage model.

2. The method of claim 1 , further comprising:
repeating the step of calculating for a second reference table of the reference tables, wherein determining comprises comparing the first and second similarity scores for determining whether the source dataset is organized as the first reference table or as the second reference table in accordance with the reference storage model.

3. The method of claim 2 , wherein the repeating is performed in response to determining that the first similarity score is smaller than a predefined similarity threshold.

4. The method of claim 1 , wherein the at least one further reference table is selected based on at least one of the further reference table has a direct relationship with the first reference table, the further reference table has an indirect relationship with the first reference table, and the number of common attributes between the source dataset and the further reference table is smaller than the number of common attributes between the source dataset and the first reference table.

5. The method of claim 1 , wherein the source dataset is received from a source system or imported from the source system or generated, wherein the source dataset comprises at least one of a table, a text file, a binary data, and a file having a hierarchical format.

6. The method of claim 1 , wherein calculating the first similarity score comprises:
calculating a partial similarity score between the source dataset and each table of the first reference table and the at least one further reference table based on respective common attributes; and
combining the partial similarity scores by weighting the partial similarity scores in accordance with the order and a direction of the relationship between the first reference table and each of the at least one further reference table.

7. The method of claim 6 , wherein the first reference table is related to the at least one further reference table via a foreign key (FK) wherein the direction indicates originating table of the first and further reference tables that comprises the primary key, wherein the order indicates the order of the FKs.

8. The method of claim 6 , wherein calculating the partial similarity score between the source dataset and the at least one further reference table comprises:
processing at least one match rule for matching values of an attribute of the source dataset with attribute values of the further reference table; and
determining a matching score indicating the level of matching between content of the source dataset and content of the further reference table, wherein the lowest the matching score the highest the partial similarity score.

9. The method of claim 1 , wherein determining a common attribute between the source dataset and the first reference table or the further reference table comprises:
determining a first attribute domain for the common attribute in the source dataset;
determining a second attribute domain for the common attribute in the first reference table; and
comparing the first and second attribute domains.

10. The method of claim 1 , wherein determining comprises comparing the first similarity score with at least one predefined similarity threshold.

11. The method of claim 1 , wherein the reference storage model provides objects, relational databases and/or indexed files.

12. The method of claim 1 , further comprising retrieving the classified source dataset.

13. The method of claim 1 , further comprising using the reference storage model and at least part of the reference tables for analyzing the source dataset if classified as the first reference table.

14. A computer program product for classifying at least one source dataset, the computer program product comprising:
a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a processor to cause the processor to provide a plurality of associated reference tables organized and associated in accordance with a reference storage model;
calculate a first similarity score between the source dataset and a first reference table of the reference tables based on common attributes in the

Assignees

• IBM

Inventors

• Oberhofer Martin
• Reddy Adapala S
• Saillet Yannick
• Seifert Jens

Classifications

• G06F16/2282

  Tablespace storage structures; Management thereof · CPC title

• G06F16/285

  Clustering or classification · CPC title

• G06F16/211 Primary

  Schema design and management · CPC title

• G06F7/20

  Comparing separate sets of record carriers arranged in the same sequence to determine whether at least some of the data in one set is identical with that in the other set or sets · CPC title

• G06F17/30292 Primary

  Physics · mapped topic