Data governance method and apparatus, and storage medium

Patent metadata

Field	Value
Publication number	US-12530328-B2
Application number	US-202418673569-A
Country	US
Kind code	B2
Filing date	May 24, 2024
Priority date	Nov 26, 2021
Publication date	Jan 20, 2026
Grant date	Jan 20, 2026

Abstract

A data governance method includes: reading a storage file from a database server, where the storage file is used to store data that belongs to at least one service; obtaining at least one data pattern of a first data set, where the first data set includes data that belongs to a same service and that is stored in the storage file, and the at least one data pattern indicates a structure of each piece of data included in the first data set; obtaining, based on the at least one data pattern, at least one data feature of the first data set; and obtaining a first data standard based on the at least one data feature.

First claim

What is claimed is:

1 . A method comprising:
reading a storage file from a database server, wherein the storage file stores data that belong to at least one service, wherein the data comprise a first data set, and wherein the first data set comprises first data that belong to a first service of the at least one service; obtaining at least one data pattern of the first data set, wherein the at least one data pattern indicates a structure of the first data; obtaining at least one data feature of the first data set based on the at least one data pattern, wherein the at least one data feature comprises a statistical feature or a distribution feature; generating, based on the at least one data feature, a first data standard regulating normativity of each piece of the first data; receiving association information associating the first data standard with the first data set in the storage file, wherein the association information comprises a file identifier of the storage file, a set identifier of the first data set, and the first data standard; and regulating, using the first data standard, normativity of to-be-stored data in the first data set.

2 . The method of claim 1 , wherein the storage file is a structured data file storing the data in a list form and the first data are stored in a same column of the structured data file, or wherein the storage file is a semi-structured data file storing the data in a label block form, and the first data are stored in a same label block of the semi-structured data file.

3 . The method of claim 1 , wherein the at least one data pattern comprises a basic data pattern indicating a basic structure of each piece of the first data, and wherein the basic structure comprises an English-digit hybrid structure, an integer structure, a floating-point number structure, a Boolean structure, an address structure, an identifier structure, or a date structure.

4 . The method of claim 3 , wherein obtaining the at least one data pattern comprises identifying each piece of the first data based on a specified regular expression to obtain the basic data pattern.

5 . The method of claim 3 , wherein the at least one data pattern further comprises a prefix data pattern indicating pieces of the first data comprising a prefix that is a longest common prefix of the plurality of the pieces.

6 . The method of claim 1 , wherein obtaining the at least one data feature comprises:
obtaining, from the first data, second data corresponding to a first data pattern of the at least one data pattern; and obtaining, based on the second data, a data occurrence quantity or a data occurrence frequency corresponding to the first data pattern, wherein the at least one data feature comprises the data occurrence quantity or the data occurrence frequency.

7 . The method of claim 1 , wherein obtaining the at least one data feature comprises:
obtaining a type of the first data based on the at least one data pattern; and obtaining the at least one data feature based on the type.

8 . The method of claim 7 , wherein the type comprises a basic type or a technology type, wherein the basic type describes a data component of the first data, and wherein the technology type describes a function implemented by the first data.

9 . The method of claim 8 , wherein the basic type is an integer type, a floating-point number type, or a Boolean type, and wherein the technology type is a code type, a coding type, a flag type, a category type, a description type, or a metric type.

10 . The method of claim 1 , wherein obtaining the at least one data feature comprises:
obtaining a statistical feature based on each piece of the first data that belongs to a first type, wherein the statistical feature comprises a maximum value, a minimum value, an average value, a deviation, a variance, a median, a percentile, or a standard deviation of each piece of the first data, wherein the first type comprises an integer, a floating-point number, a metric type or a coding type, and wherein the at least one data feature comprises the statistical feature; or obtaining a distribution feature based on each piece of the first data that belongs to a second type, wherein the distribution feature comprises an occurrence quantity or an occurrence frequency of each piece of the first data, wherein the second type comprises a flag type, a Boolean type, a category type, or a code type, and wherein the at least one data feature comprises the distribution feature.

11 . The method of claim 1 , wherein the at least one data feature comprises a segmented word, wherein the method further comprises further obtaining the at least one data feature by obtaining the segmented word by performing segmentation on second data and removing a stop word from the second data, and wherein the second data comprise service attribute description information corresponding to the first data set or third data of a description type in the first data set.

12 . The method of claim 11 , wherein the at least one data feature comprises a data occurrence quantity or a data occurrence frequency corresponding to a language, and wherein the third data belong to the language.

13 . The method of claim 1 , wherein obtaining the first data standard comprises:
determining, based on the at least one data feature, a connected graph to which the first data set belongs, wherein nodes in the connected graph are different data sets of the data, and wherein a similarity between a neighboring node of the first data set and a node of the first data set exceeds a specified threshold; and obtaining the first data standard based on a data set in the connected graph.

14 . The method of claim 13 , further comprising further determining the connected graph based on a service type of the first data, wherein the service type is a date, a region, an address, or an identifier.

15 . The method of claim 13 , wherein obtaining the first data standard comprises:
obtaining at least one data standard comprising a data standard associated with another data set other than the first data set; and selecting the first data standard from the at least one data standard based on an association frequency of each data standard in the at least one data standard.

16 . An apparatus comprising:
a memory configured to store an instruction; and one or more processors configured to execute the instruction to cause the apparatus to:
read a storage file from a database server, wherein the storage file stores data that belong to at least one service, wherein the data comprise a first data set, and wherein the first data set comprises first data that belong to a first service of the at least one service;
obtain at least one data pattern of the first data set, wherein the at least one data pattern indicates a structure of the first data;
obtain at least one data feature of the first data set based on the at least one data pattern, wherein the at least one data feature comprises a statistical feature or a distribution feature;
generate, based on the at least one data feature, a first data standard regulating normativity of each piece of the first data;
receive association information associating the first data standard with the first data set in the storage file, wherein the association information comprises a file identifier of the storage file, a set identifier of the first data set, and the first data standard; and
regulate, using the first data standard, normativity of to-be-stored data in the first data set.

17 . The apparatus of claim 16 , wherein the storage fi

Assignees

• Huawei Cloud Computing Tech Co Ltd

Inventors

• Hao Shiyuan
• Wei Ziheng
• Long Jiang
• Lu Hong
• Ji Zhenfeng

Classifications

• G06F16/2365

  Ensuring data consistency and integrity · CPC title

• G06F16/258

  Data format conversion from or to a database · CPC title

• G06F16/84

  Mapping; Conversion · CPC title

• G06F16/215 Primary

  Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors · CPC title

• G06F16/86

  Mapping to a database · CPC title